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ABSTRACT 


Recently,  a  great  deal  of  attention  has  been  given  to  the  use  of  Massively 
Multiplayer  Online  Games  (MMOGS)  for  both  gaming  and  military  applications.  The 
revenue  generated  by  MMOGs  and  the  effect  that  they  have  on  the  network  infrastructure 
has  resulted  in  significantly  more  developmental  resources  being  applied  to  commercial 
MMOG  technology  than  for  military  distributed  virtual  (DVE)  development.  All  DVEs 
share  a  common  set  of  characteristics,  and  additional  requirements  exist  for  the 
interoperability  of  military  DVEs.  It  is  possible  to  exploit  these  similarities  to  take 
advantage  of  developments  in  the  supporting  technologies  of  commercial  MMOGs. 

Specific  capabilities  of  interest  include  scalability  for  large  numbers  of  players, 
capacity  for  large  amounts  of  network  traffic,  portability  across  operating  systems,  and 
adaptability  to  connect  diverse  codebases,  network  protocols,  and  data  formats.  Project 
Darkstar  is  a  Sun  Eabs  research  project  that  has  developed  an  open-source  middleware 
for  MMOGs.  This  thesis  has  produced  and  tests  a  MMOG  server,  which  interconnects 
heterogeneous  simulators  in  a  DVE  using  the  Project  Darkstar  middleware  and  locally 
developed  network  gateways.  The  performance  of  the  system  and  the  character  of  the 
network  traffic  it  generates  are  analyzed.  Initial  test  results  warrant  further  development 
and  eventual  deployment. 
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I.  INTRODUCTION 


A,  PROBLEM  AND  THESIS  STATEMENT 

Throughout  the  history  of  games,  there  has  been  interplay  between  gaming  for 
military  purposes  and  gaming  for  entertainment  purposes  (Smith,  2009;  Zyda,  1997). 
Attention  has  lately  been  direeted  towards  finding  military  applieations  for  Massively 
Multiplayer  Online  Games  (MMOG)  (Bonk  &  Dennen,  2005).  MMOGs  eontinue  to  be 
one  of  the  fastest  growing  segments  of  the  eomputer  game  market.  Typieally,  the 
eommereial  MMOG  environment  is  dominated  by  role-playing  games  (RPG).  This  is  a 
funetion  of  the  eeonomies  of  eommereial  MMOG  development,  sales  and  operation. 
There  is  nothing  inherent  in  MMOG  architeeture  that  requires  them  to  be  RPGs.  The 
inherent  eharaeteristies  of  MMOGs  are  sealability,  persistenee,  and  a  eonsistent  world 
state. 

For  purposes  of  this  thesis,  the  term  Distributed  Virtual  Environment  (DVE)  is 
used  to  mean  a  sealable,  virtual  environment  that  enforees  to  some  degree  a  eommon 
world  state  for  all  elients  independent  of  a  speeifie  session.  Several  important  questions 
relate  to  use  of  MMOG  teehnology  in  military  simulations.  What  state  information  needs 
to  be  made  available  to  partieipants  in  this  environment?  Is  it  possible  to  leverage  the 
investment  in  existing  simulators  by  adapting  them  to  funetion  as  the  user-interfaee  for  a 
persistent  virtual  environment?  Can  improved  performanee  or  additional  functionality  be 
gained  by  using  Distributed  Interactive  Simulation  (DIS)  protocol  and  MMOG  hybrid 
architecture?  These  questions  are  examined  in  this  thesis. 

B.  MOTIVATION 

The  military  and  business  communities  have  long  noted  the  potential  for 
simulation  and  gaming  technology  to  develop  high-order  thinking  skills.  It  has  been 
suggested  that  certain  skills  gained  and  practiced  by  gamers  in  multiplayer  online  gaming 
environments  closely  parallel  those  required  by  teams  of  operators  interacting  in  the  real 
world.  This  analogy  might  be  pushed  even  further  to  that  of  a  military  transforming  to 
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operate  under  the  coneept  of  network-eentrie  warfare.  A  DVE  that  is  able  to  maintain  an 
objeetive  state  independent  of  whether  any  participants  are  connected  to  it  may  lend  itself 
to  medium-term  and  long-term  virtual  exercises.  Sufficient  scalability  is  needed  to  allow 
large  numbers  of  participants.  The  growth  and  popularity  may  mean  that  technology  to 
facilitate  development  and  operation  of  DVE  for  military  applications  might  be  made 
rapidly  available  by  taking  advantage  of  the  competition  within  the  gaming  industry. 

It  is  important  to  note  that  a  DVE  and  an  MMOG  are  not  necessarily  the  same.  By 
way  of  distinction,  a  game  may  be  described  as  a  real-time  system  having  actors,  goals, 
rules,  feedback  in  the  form  of  a  score,  and  some  sort  of  plot  or  scenario.  A  DVE  as 
described  here  may  only  possess  some  of  these  characteristics.  A  DVE  intended  to  be 
used  for  training  and  experimentation  may  only  need  to  support  hundreds  of  participants, 
not  the  tens  or  hundreds  of  thousands  of  linked  players  that  commercial  MMOGs  must 
support  in  order  to  be  economically  viable.  Given  the  scope  of  today’s  problems  of 
interest,  however,  hundreds  of  participants  are  hardly  “Massive.”  Explicit  user  interaction 
within  a  DVE  is  usually  defined  at  the  client  level  by  the  application  interface.  Such 
interaction  can  range  from  text  messages  from  a  mobile  phone  to  immersive  stereo  and 
data  gloves  in  a  3D  virtual  world. 

Currently,  the  IEEE  Distributed  Interactive  Simulation  (DIS)  network  protocol  is 
widely  used  to  enable  interoperability  for  distributed  simulation  in  military  applications. 
In  order  to  deliberately  move  from  known  to  projected  DVE  capabilities,  this  thesis  work 
first  takes  advantage  of  the  rich  set  of  DIS-enabled  X3D  models  provided  by  the  online 
Scenario  Authoring  and  Visualization  for  Advanced  Graphical  Environments  (Savage) 
and  Eor  Official  Use  Only  (EOUO)  SavageDefense  model  archives.  The  ongoing 
production  of  agent-based  and  robot-based  models  that  are  completely  visualized  using 
X3D  provides  a  large  and  growing  set  of  exemplars.  Prior  thesis  work  comparing 
prominent  DVEs  (Sanders,  2008)  provides  a  good  summary  of  baseline  capabilities. 
Relevant  thesis  work  regarding  metadata  for  distinguishing  entities  is  SAVAGE 
Modeling  and  Analysis  Language  (SMAL)  (Rausch,  2006). 
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It  is  also  possible  to  develop  a  loeal  DIS  interfaee  to  the  SH-60B  MRT3  training 
deviee  and  other  simulations  that  enable  eommunieation  with  a  persistent  virtual 
environment.  Sueh  an  interfaee  is  developed,  deseribed  and  demonstrated  in  this  thesis. 
This  proof  of  eoneept  demonstrates  that  any  real-time  training  or  taetieal  deviee,  whieh 
uses  DIS  for  interoperability,  is  also  able  to  partieipate  in  the  DVE.  Additional  data 
formats  and  network  protoeols  with  semantieally  similar  information  ean  be  then  as 
added  as  eonneetions  to  this  MMOG  server. 

C.  SCOPE  OF  THE  THESIS 

The  thesis  is  limited  to  an  exploration  of  the  requirements  for  a  DVE  to  support 
distributed  training  and  experimentation.  This  requirements  set  is  used  to  develop  a 
reeommended  arehiteeture  for  sueh  a  DVE.  A  small-seale  DVE  is  developed  using 
Projeet  Darkstar®  as  middleware.  Eessons  learned  from  this  demonstration  are  then  used 
to  develop  a  DVE,  whieh  is  eapable  of  supporting  four  or  more  separate  and  remote 
generie  deviees.  Eoeal  (elient)  interfaees  are  ereated  for  the  SH-60B  MRT3  and  a 
eommereial  desktop  based  flight  simulator.  A  visualization  of  the  environment  state  is 
eonstrueted  using  the  X3D  Graphies  open  standard  for  defining  and  eommunieating  real¬ 
time,  interaetive  3D  eontent.  Test  results  are  added  to  the  Savage  model  arehives  and 
software  extensions  are  added  to  the  online  open-souree  Open-DIS  eodebase.  Einally,  the 
overall  system  is  demonstrated  and  a  performanee  test  of  the  system  sealability  is 
eondueted. 

D,  THESIS  ORGANIZATION 

The  remainder  of  this  thesis  is  organized  as  follows. 

1.  Chapter  II:  Background 

This  ehapter  provides  a  general  funetional  deseription  of  distributed  virtual 
environments  with  a  foeus  on  enabling  interoperability  and  interaetivity  for  distributed 
partieipants.  Existing  standards  for  distributed  interaetive  applieations  are  briefly 
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examined  and  an  argument  is  presented  for  the  relevanee  of  researeh,  whieh  seeks  to 
apply  MMOG  arehiteetures,  and  middleware  to  military  distributed  virtual  environments 
(DVEs). 

2.  Chapter  III:  Related  Work 

This  ehapter  examines  previous  work  speoifieally  related  to  alternative 
arehiteetures  for  distributed  virtual  environments  ineluding  MMOGs  and  applieation 
middleware.  This  ehapter  also  presents  an  overview  of  the  existing  Modeling  & 
Simulation  (M&S)  applieations  used  as  test  eases  for  the  implementation  portion  of  this 
work. 


3,  Chapter  IV:  Methodology 

Details  of  the  implementation  ehoiees  relevant  to  this  DVE-eonstruetion  work  are 
presented  here.  In  addition  to  implementation  of  an  MMOG  server,  the  use  of 
heterogeneous  simulation  applieations  required  the  eonstruetion  of  several  gateways. 
Development  of  an  open-souree  gateway  for  a  high  frame -rate  eommereial  flight 
simulator  is  also  presented.  Sinee  a  virtual  environment  is  not  eomplete  without 
visualization,  the  implementation  of  a  graphieal  refleetion  of  the  virtual  environment  state 
using  open-souree,  open-standards  tools  is  presented.  Methods  for  measuring  the 
performanee  of  the  system  arehiteeture  are  also  given. 

4,  Chapter  V:  Results,  Performance  and  Behavior 

Quantitative  and  qualitative  system  performanee  is  presented  in  this  ehapter. 
Software  profiling  data  of  the  server  under  different  loads  is  presented  as  well  as  network 
analysis  of  the  eommunieation  between  the  server  and  a  elient  simulator. 

5,  Chapter  VI:  Conclusions  and  Recommendations 

Based  upon  the  insight  gained  from  this  work,  eonelusions  are  presented  on  the 
feasibility  of  using  MMOG  arehiteeture  for  a  military  distributed  virtual  environment.  In 
addition  to  the  funetionality  demonstrated  in  this  work,  several  other  useful  eapabilities 
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may  be  gained  with  this  hybrid  arehiteeture.  This  work  is  expected  to  serve  as  the 
foundation  for  a  dedicated  Massively  Multiplayer  Online  (MMO)  service  hosted  at  the 
Naval  Postgraduate  School. 
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II.  BACKGROUND 


A,  INTRODUCTION 

The  DoD  pioneered  the  use  of  Distributed  Virtual  Environments  (DVEs)  for  the 
training  of  large  numbers  of  heterogeneous  interaeting  units  with  the  development  of 
Simulator  Networking  (SIMNET).  The  design  ehoiees  made  for  SIMNET  sill  heavily 
influenee  the  development  of  DVEs  today.  Whether  they  are  eomplex  networks  of 
military  training  deviees  or  entertainment  serviees  sueh  as  Massively  Multiplayer  Online 
Games  (MMOGs),  all  DVEs  share  a  eommon  set  of  features.  One  key  feature  is  the 
ability  of  eaeh  partieipant  to  have  a  shared  sense  of  state  of  the  environment.  Due  to  some 
fundamentally  diffieult  eharaeteristies  of  heterogeneous  distributed  systems,  assuring 
suecess  is  an  inherently  complex  and  demanding  task.  The  growth  of  commercial 
MMOGs  means  that  significant  resources  outside  of  the  DoD  are  being  invested  and 
applied  to  addressing  similar  challenges,  and  the  technology  in  the  MMOG  sector  will 
continue  to  develop  more  rapidly  than  the  corresponding  technology  in  the  defense 
sector.  It  may  therefore  be  possible  to  take  advantage  of  this  development  with  DVE 
architectures  that  are  hybrids  of  current  standards  and  emerging  technologies. 

B,  DISTRIBUTED  VIRTUAL  ENVIRONMENTS 

The  scenario  is  dramatic.  The  group  of  tanks  swiftly  moves  cross-country  to  cut 
off  the  opposing  armored  column.  It  is  late  afternoon  but  the  overcast  and  the  dust  reduce 
visibility  to  almost  nighttime  conditions.  The  tank  commander  in  the  lead  vehicle  peers 
through  his  night  vision  scope  trying  to  make  out  the  shapes  on  the  distant  rise  ahead.  A 
quick  verbal  command  to  the  gunner  and  the  turret  slews  right.  There,  just  moving  from 
behind  an  earthen  revetment  is  definitely  a  tank,  no  three  tanks.  He  calls  out  a  sighting 
report  as  the  gunner  prepares  to  fire.  Soon  the  rest  of  the  regiment  would  follow.  The 
Battle  of  “73  Easting”  had  begun. 
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The  above  vignette  is  direetly  relevant  to  the  subjeet  of  distributed  virtual 
environments  beeause  one  of  the  first  sueeessful  distributed  virtual  environments  (DVE), 
SIMNET,  was  used  to  augment  the  training  of  the  Armored  Cavalry  Regiment 
soldiers  that  fought  in  that  one-sided  battle  on  Eebruary  26,  1991.  The  SIMNET 
infrastrueture  was  also  used  to  reereate  the  battle  for  purposes  of  visualization  and 
analysis  after  the  fact  (Eigure  1).  The  architectural  and  conceptual  decisions  made  for  the 
implementation  of  SIMNET  still  heavily  influence  the  development  of  DVEs  today 
(Eenoir,  2003;  Miller  &  Thorpe,  1995).  A  functional  definition  is  needed  for  the  term 
distributed  virtual  environment,  identifying  the  primary  factors,  which  constrain  their 
scale  and  consistency. 


Eigure  1.  (left)  SIMNET  vehicle  simulators  at  Eort  Knox,  KY.  (right)  SIMNET 

screenshot  Erom  Bruce  Sterling's  “War  is  Virtual  Hell,”  (Erom;  Sterling, 
1993). 

Military  operations  by  their  nature  involve  many  heterogeneous  interacting  units. 
Historically,  due  to  system  complexity  and  hardware  costs,  as  well  as  the  benefits  gained 
by  collective  training,  the  military  has  been  the  leading  pioneer  in  the  development  of 
distributed  virtual  environments  (Miller  &  Thorpe,  1995).  As  hardware  performance  has 
improved  to  the  point  where  even  desktop  computers  can  outperform  the  expensive 
purpose-built  systems  of  only  a  few  years  ago,  and  as  relatively  high-bandwidth  network 
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infrastructure  has  become  ubiquitous  beyond  the  military  domain,  the  commercial  game 
industry  (with  some  significant  caveats)  is  now  leading  the  development  of  DVE 
architectures  (Smith,  2009;  Narayanasamy  et  ah,  2006). 

Regardless  of  whether  the  system  is  a  tool  for  combined  arms  training  such  as 
SIMNET  or  a  Massively  Multiplayer  Online  Game  (MMOG)  such  as  World  of 
Warcraft®,  all  DVEs  can  be  described  as  a  system,  which  possesses  the  five  following 
features. 

•  A  shared  sense  of  space:  All  participants  have  the  illusion  of  being  in 
some  physical  relation  to  one  another  whether  adjacent  or  across  a  field. 

•  A  shared  sense  of  presence:  Participants  take  on  a  persona  or  avatar  when 
they  enter  and  interact  within  this  environment.  The  avatar  is  not 
necessarily  a  human. 

•  A  shared  sense  of  time:  Some  degree  of  real-time  interaction  and  temporal 
consistency  must  occur. 

•  A  way  to  communicate:  Voice  or  message  transmission  among 
participants  adds  a  necessary  sense  of  realism  to  any  simulated 
environment. 

•  A  way  to  share:  The  ability  to  interact  with  other  participants  in  the 
environment  includes  the  ability  to  have  a  shared  state  of  the  environment 
itself  (Singhal  &  Zyda,  1999). 

In  addition,  four  basic  components  are  required  to  implement  a  DVE. 

•  Graphics  engines  and  displays 

•  Communications  and  control  devices 

•  Processing  Systems 

•  Data  Network  (Singhal  &  Zyda,  1999). 

As  a  distributed  computer  system,  at  its  most  abstract  a  DVE  is  a  system  for 
passing  messages  between  computers  over  a  network.  DVE  designers  and  developers 
must  cope  with  the  challenges  of  managing  network  resources,  dealing  with  concurrency, 
and  compensating  for  the  variability  in  the  transport  of  network  traffic  (Diehl,  2001).  In 
an  idealized  shared  virtual  environment,  all  participants  need  to  have  completely  up-to- 
date  knowledge  of  the  environment  state  and  also  need  to  interact  with  the  environment 


9 


and  other  participants  in  a  natural  and  seamless  manner.  As  a  specific  example,  if 
participant  A  picks  up  an  object,  participant  B  (within  the  limits  of  human  or  game 
perception)  needs  to  have  knowledge  of  this  change  of  environment  state. 

Network  latency  is  defined  as  the  amount  of  time  required  to  transfer  a  data 
message  from  one  point  to  another.  In  a  Local  Area  Network  (LAN),  this  is  on  the  order 
of  10  milliseconds  (ms).  It  is  on  the  order  of  100  ms  for  transcontinental  transfer  and  250 
to  500  ms  for  intercontinental  transfer  (Pullen,  2000;  Singhal  &  Zyda,  1999).  The 
theoretical  lower  limit  is  based  on  the  speed  of  light,  approximately  10  ms  per  time  zone. 
Due  to  network  latency  and  processing  delays,  it  is  theoretically  impossible  to  have  a 
highly  dynamic  shared  state  and  high  consistency  at  the  same  time.  This  is  referred  to  as 
the  consistency-throughput  tradeoff. 

Consistency-Throughput  Tradeoff:  It  is  impossible  to  allow  dynamic 
shared  state  to  change  frequently  and  guarantee  that  all  hosts 
simultaneously  access  identical  versions  of  that  state.  (Singhal  &  Zyda, 

1999) 

Consistency  in  this  context  is  best  defined  as  information  uniformity  among  the 
parts  of  a  complex  system.  Throughput  is  the  rate  at  which  such  shared  state  is  updated. 
While  graphics  engines  and  computer  processing  may  impose  some  constraint  on  how 
quickly  updates  to  shared  state  may  be  displayed,  it  is  the  architecture  of  the  DVE  and  the 
characteristics  of  the  network  itself  that  is  primarily  the  cause  of  the  consistency- 
throughput  tradeoff  (Singhal  &  Zyda,  1999;  Pelligrino  &  Dovrolis,  2003;  Tannenbaum  & 
Steen,  2002). 

In  general,  DVEs  place  a  significant  demand  on  network  resources.  Without 
modification,  the  original  SIMNET  architecture  would  require  375  Mbps  of  network 
bandwidth  for  100,000  players  (Macedonia,  1995).  Modern  MMOGs,  which  are 
relatively  sparse  with  the  amount  of  data  that  is  exchanged  in  each  message,  still  require 
approximately  2-3  Kbps  of  bandwidth  per  player  (Chen  et  ah,  2005).  This  is  actually 
comparable  to  the  bandwidth  per  participant  required  for  military  DVEs  (Keune  & 
Coppock).  Commercial  and  military  DVE  developers  face  similar  challenges  and 
consequentially  are  forced  to  consider  similar  solutions. 
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Most  commercial  distributed  virtual  environments  use  a  client-server  architecture 
in  order  to  maintain  authority  over  the  interactions  between  participants  (Hall  &  Novak, 
2008;  IGDA,  2004;  Bogojevic  &  Kazemzadeh,  2003).  This  includes  administrative 
interactions  such  as  authentication  and  session  matchmaking,  or  simulation  interactions 
such  as  collision  detection  and  combat  resolution.  The  server  manages  the  current  state  of 
the  entire  world.  A  client  sends  an  update  to  the  server,  which  propagates  it  to  all  other 
clients.  When  all  communication  is  via  the  server,  the  server  is  a  potential  bottleneck.  In 
peer-to-peer  architectures,  every  client  has  a  partial  copy  of  the  state  of  the  world.  If 
something  changes,  the  client  has  to  send  this  change  to  all  other  clients.  Client-server 
and  peer-to-peer  are  the  extremes  of  the  spectrum  of  possible  architectures  (Figure  2).  For 
large  flexible  virtual  worlds,  hybrids  are  often  more  appropriate  (Hsu  et  ah,  2003;  Diehl, 
2001). 


(a)  Client-Server  Architecture 


(b)  Peer-to  Peer  Architecture 


(c)  Multi-Server  Architecture 


Figure  2.  Topologies  of  Distributed  Virtual  environments.  In  practice,  hybrid 

topologies  are  often  used  for  large  DVEs  in  the  military  domain.  (From;  Hsu 
et  ah,  2003) 


The  basic  functional  characteristics  of  a  distributed  virtual  environment  have  been 
examined.  One  of  the  primary  components,  the  network  infrastructure,  is  generally 
beyond  the  direct  control  of  the  developer.  The  consequence  is  a  fundamental  limitation 
on  a  DVEs  capability  as  described  by  the  consistency- throughput  tradeoff.  Generic 
architectures  for  managing  consisteney  and  throughput  usually  have  common 
architectural  designs. 


11 


C.  LEVERAGING  THE  DEVELOPMENT  OF  MASSIVE  MULTIPLAYER 

ONLINE  GAMES  (MMOGS)  FOR  DISTRIBUTED  SIMULATION 

A  massively  multiplayer  online  game  (MMOG)  is  a  distributed  virtual 
environment  (DVE)  in  whieh  a  large  number  of  people  ean  partieipate  simultaneously 
over  a  network,  internet  with  each  other  (and  usually  interact  with  the  game  world),  join 
in  or  leave  at  any  time  and  expect  everything  they  have  produced  to  persist  while  they  are 
offline  (Hall  &  Novak,  2008).  Applications  may  vary  from  simple  text-based  Multi-User 
Dungeons  (MUDs)  to  visually  rich  role-playing  games.  Both  extremes  meet  the 
functional  requirements  of  a  DVE  that  were  presented  above.  While  there  has  always 
been  interest  in  the  cognitive  and  social  dynamics  of  participant  interaction  within  shared 
virtual  environments,  recently  there  has  been  an  explosion  in  academic  and  commercial 
interest  regarding  the  development  of  both  the  technology  to  enable  these  environments 
and  the  content  to  make  them  viable  (Bonk  &  Dennen,  2005).  The  rapid  growth  in  the 
number  of  subscribers  has  consequences  that  extend  beyond  the  aesthetic  value  or  social 
implications  of  interacting  with  large  numbers  of  other  participants.  Commercial  MMOG 
development  is  being  driven  fundamentally  because  of  the  fact  that  when  they  are 
successful,  they  can  be  much  more  profitable  than  single-player  computer  games  (Hall  & 
Novak,  2008).  As  the  number  of  subscribers  grows  (Eigure  3),  there  is  a  corresponding 
push  to  develop  the  technology  required  to  manage  the  large  numbers  of  participants 
within  individual  environments,  corresponding  efforts  are  also  needed  to  manage  the 
significant  and  growing  impact  that  the  message  traffic  of  these  game  has  on  the  internet 
infrastructure.  Developers  of  DVEs  for  military  applications  must  both  carefully  watch 
both  trends  and  actively  participate  in  this  process  of  innovation. 

Eor  a  premier  single-player  computer  game  to  be  profitable,  where  development 
costs  range  from  $15  to  $20  million,  500,000  unit  sales  is  the  minimum  needed.  Some 
have  argued  that  the  computer  game  market  can  no  longer  support  development  of  more 
than  a  few  such  titles  (Deering,  2009).  Even  though  first-class  MMOGs  may  cost  more  to 
develop,  often  they  can  become  profitable  at  only  250,000  unit  sales.  MMOGs  can  also 
lose  much  more  money  because  of  the  infrastructure  cost  of  supporting  the  game  (Hall  & 
Novak,  2008).  Per-subscriber  revenue  for  commercial  MMOGs  averages  between  $9.95  a 
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month  and  $12.95  a  month  (Tay,  2005;  IGDA,  2004).  A  naive  analysis  shows  that  an 
MMOG  that  maintains  an  average  of  250,000  subseribers  over  five  years  will  generate  on 
the  order  of  $150  million  in  revenue.  Extremely  sueeessful  MMOGs  like  World  of 
Warcraft®  might  generate  this  amount  of  revenue  in  a  few  months  (Lent,  2008).  The  cost 
of  operating  and  updating  this  ongoing  service  must  also  be  factored  into  the  budget 
equation.  Arguably,  this  post-sale  revenue  potential  is  the  key  economic  factor,  which  is 
driving  the  development  of  MMOG  technology. 


MMOG  Active  Subscriptions 

200,000+ 


Ultima  Online 

Lineage 

EverQuest 

Dark  Age  of  Camelot 

RuneScape 

Final  Fantasy  XI 

EVE  Online 

Star  Wars  Galaxies 

Lineage  II 

Dofus 

EverQuest  II 

World  of  Warcraft 

The  Lord  of  the  Rings  Online 


Figure  3.  MMOG  Active  Subscriptions;  (http://www.mmogchart.com).  This  shows 

the  growth  of  MMOGs  and  the  dominance  of  the  top  six.  (From;  Woodcock, 
2008) 


As  the  number  of  subscribers  grows  rapidly,  it  follows  that  the  number  of 
simultaneous  users  will  grow  as  well.  In  general,  distributed  virtual  environments  place 
great  demands  on  network  resources,  and  such  demands  are  expected  to  increase 
(Claypool,  2005).  For  example,  Gamania,  a  Taiwanese  company  that  operates  the  game 
Lineage,  owns  more  than  4,000  Mbps  of  dedicated  links  for  game  traffic  (Chen  et  ah. 
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2005).  This  imposes  a  cost  on  the  providers  of  MMOGs  and  has  an  impact  on  other  users 
of  the  network.  This  is  a  real  constraint  on  the  growth  potential  of  these  environments.  As 
has  been  discussed,  the  Internet,  while  generally  extremely  successful  at  scaling  up  to  a 
global  sized  system,  was  not  designed  with  distributed  virtual  environments  in  mind.  As 
these  games  grow,  they  are  likely  to  have  an  influence  on  the  way  that  network 
infrastructure  develops. 

The  development  software  required  to  implement  such  games  has  followed  a 
traditional  arc,  from  closed  and  proprietary  software  tied  to  a  single  game,  to  commercial 
toolkits  decoupled  from  a  specific  game,  to  multiple,  disparate  open-source 
implementations.  As  the  segment  further  develops,  a  nascent  standardization  and 
interoperability  effort  is  beginning  to  coalesce  in  the  Internet  Engineering  Task  Force 
(IETF)  space  (MMOX  Wiki),  primarily  around  architecture  closely  related  to  Einden 
Eabs’  Second  Fife.  The  maturation  of  the  commercial  MMOG  market  creates  an 
opportunity  for  the  DoD.  If  a  viable,  standards-based  community  develops  around 
MMOGs,  the  DoD  may  be  able  to  piggyback  off  this  effort  and  exploit  the  economies  of 
scale,  commercial  tools,  and  base  of  experienced  engineers  produced  by  the  commercial 
game  industry  (Bonk  &  Dennen,  2005).  However,  this  opportunity  must  be  balanced 
against  the  requirements  of  a  large  installed  base  of  legacy  M&S  software. 

D,  ENABLING  INTEROPERABILITY  IN  DISTRIBUTED  VIRTUAL 

ENVIRONMENTS 

Interoperability  is  the  ability  to  connect  simulation  clients  together  such  that  those 
clients  can  operate  on  a  perceived  shared  virtual  environment  (Zyda,  2005).  A  useful 
distributed  virtual  environment  needs  to  enable  heterogeneous  hardware  and  software 
applications  to  participate  in  the  environment.  This  is  especially  critical  for  military 
DVEs  since  there  has  been  a  great  deal  of  investment  made  in  hardware  and  software 
without  consideration  to  DVE  interoperability.  Subsequently,  much  effort  has  been 
expended  to  enable  interoperability  in  order  produce  a  common  synthetic  environment 
(Page,  2002).  This  section  examines  both  the  syntactic  and  semantic  interoperability 
prerequisites  for  the  use  of  heterogeneous  hardware  and  software  in  a  DVE.  This  is  a 
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fundamentally  difficult  problem.  Interoperability  can  also  be  described  as  the  meaningful 
exchange  of  information  between  participants  in  a  DVE.  The  Distributed  Interactive 
Simulation  (DIS,  IEEE  1278)  and  High  Eevel  Architecture  (HEA,  IEEE  1516)  are 
discussed  and  the  ability  of  each  standard  to  enable  interoperability  is  briefly  examined. 
The  architecture  of  MMOGs  is  reviewed  in  general  and  posited  as  an  alternative 
architecture  for  military  DVEs.  Einally,  the  necessary  quality  of  this  information 
exchange  is  discussed  with  specific  attention  to  update  rate  and  latency  for  shared  state. 

1.  Motivation  for  Interoperability 

Interoperability  is  a  fundamental  requirement  of  military  virtual  environments 
because  military  operations  are  highly  complex  and  heterogeneous.  Nevertheless 
military  simulations  tend  to  be  developed  for  specific  purposes.  In  addition,  the  idea  of 
cost  savings  through  reusability  and  composability  has  driven  the  desire  for 
interoperability  among  DoD  virtual  environment  applications  (Tolk  &  Muguira,  2003; 
Davis  &  Anderson,  2003).  Eurther  significant  benefits  can  occur  if  tactical  systems 
become  capable  of  interoperations  with  simulations  systems  (Brutzman  et  ah,  2002). 

DoD  virtual  environment  applications  range  from  relatively  simple  PC-based 
simulations  to  multi-million  dollar  training  devices.  Although  often  a  great  deal  of  effort 
and  expertise  has  gone  into  their  development,  generally  speaking  each  is  developed  for  a 
specific  purpose  without  considering  for  the  need  to  interact  in  a  meaningful  manner  with 
one  another.  The  fact  that  military  operations  are  inherently  highly  complex  and 
heterogeneous  means  that,  in  order  to  use  these  virtual  environments  in  a  way  that 
reflects  real  world  military  operations,  each  must  be  composed  coherently  into  a 
distributed  virtual  environment  (Miller  &  Thorpe,  1995). 

The  complexity  of  developing  monolithic  virtual  environment  systems  has  often 
led  to  failures  in  system  development.  A  possible  alternative  to  monolithic  systems  is 
composability,  which  is  the  selection  and  assembly  of  previously  existing  components  to 
satisfy  some  user  requirement.  Interoperability  is  a  necessary  precondition  for 
composability  (Page  et  ah,  2002).  If  two  systems  cannot  exchange  information  in  a 
meaningful  manner  then  they  certainly  cannot  interact  to  satisfy  overall  system 
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requirements  (Davis  &  Anderson,  2003).  Therefore,  eomposability  reduees  the  risk  of 
developing  large  seale  virtual  environment  systems  and  leverages  the  huge  investment  in 
previously  existing  systems.  Sueh  eomposability  beeomes  far  more  powerful  if  the 
funetional  improvements  are  foeused  on  data-stream  interoperability  rather  than  eodebase 
re-engineering. 

2.  Theory  and  Practice  of  Interoperability 

Interoperability  is  defined  in  general  as: 

The  ability  of  systems,  units  or  forees  to  provide  serviees  to  and  aeeept 
serviees  from  other  systems,  units  or  forees,  and  to  use  the  serviees  so 
exehanged  to  enable  them  to  operate  effeetively  together.  (Joint 
Publieation  1-02) 

A  teehnologieal  definition  of  interoperability  may  be  as  simple  as  the  ability  to 
exehange  data  or  serviees  at  run-time.  For  distributed  virtual  environments  this  definition 
must  be  extended  beeause  individual  simulations  whether  they  are  game-based  trainers  or 
full-motion  flight  simulators  must  aet  on  this  data  in  order  for  a  shared  virtual 
environment  to  exist.  While  intereonneetion  as  a  prerequisite  for  interoperability,  there  is 
little  point  to  intereonneeted  eommunieation  systems  if  the  meaning  of  the  exehanged 
information  is  different  in  eaeh  system. 

A  great  deal  of  rhetorie  has  been  published  regarding  to  the  notion  of  plug  and 
play  interoperability  of  different  simulations.  The  aetual  experienee  of  eomposing  large 
numbers  of  heterogeneous  simulations  together  has  foreed  many  users  and  developers  of 
distributed  virtual  environments  to  re-examine  the  motivation  and  theoretieal  foundations 
for  arbitrary  simulator  interoperability  (Ceranowiez  et  ah,  2002;  Davis  &  Anderson, 
2003).  Reeent  work  has  generally  eoneluded  that  it  is  impossible  to  apply  a  fixed 
standard  of  interoperability  aeross  all  domains  of  modeling  and  simulation,  and  that  a 
level-of- interoperability  metrie  might  better  be  used  instead  (Tolk  &  Muguira,  2003).  A 
fundamental  definition  of  a  model  is  representation  of  an  element  of  the  real  world  for  a 
purpose.  A  simulation  is  the  eorresponding  exeeution  and  response  over  time.  By  the 
nature  of  model  eonstruetion,  there  are  usually  explieit  and  (more  importantly)  implieit 
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assumptions  made  that  reveal  gaps  when  the  model  is  eompared  to  the  real  world.  In 
attempting  to  enable  these  models  to  interaet  with  real  world  systems  or  other  models, 
there  is  a  tendeney  towards  complexity.  As  model  complexity  increases,  it  soon  reaches  a 
point  where  the  risk  of  failure  rises  much  more  quickly  (Chwif  et  ah,  2000). 

Distributed  virtual  environments  (DVEs)  are  directly  analogous  to  command  and 
control  (C2)  systems  and  distributed  databases  because  they  are  all  essentially  means  of 
sharing  a  common  state.  The  notion  of  information-exchange  requirements  (lERs)  is  one 
of  the  central  concepts  in  the  development  of  these  types  of  systems  (Tolk  &  Turnitsa, 
2007).  Eor  distributed  virtual  environments,  the  lERs  can  be  summarized  as  follows. 

•  Syntactic:  data  type,  message  length,  protocol 

•  Contextual/Conceptual:  modeling  assumptions,  purpose  of  the  information 

•  Semantic:  units  and  meaning  of  data,  conventions  for  internal  algorithms 

Syntactic  interoperability  is  primarily  an  engineering  problem  and  is  generally 
addressed  with  well-defined  and  broadly  accepted  standards.  Elnderstanding  of 
conceptual  interoperability  is  perhaps  incomplete  but  there  is  usually  sufficient 
conceptual  understanding  to  enable  correct  syntactic  mappings  between  different  data 
formats.  Semantic  interoperability  is  achieved  by  having  a  common  method  of 
identifying  data,  such  as  a  common  data  model  (Davis  &  Anderson,  2003;  Tolk  & 
Muguira,  2003).  Arguably  DIS  and  other  network  protocols  completely  address  syntactic 
interoperability.  To  some  extent  both  DIS  and  HEA  address  semantic  interoperability  as 
it  is  defined  above.  One  important  element  of  semantic  interoperability  is  the  need  for  a 
common  way  of  specifying  location  as  some  models  may  use  geocentric  Cartesian  (x,  y, 
z)  and  others  spherical  (latitude,  longitude,  and  altitude)  coordinate  systems.  Middleware, 
distributed  algorithms,  and  data-type  coercion  are  generally  used  to  solve  the  problem  of 
semantic  interoperability.  Middleware  is  a  software  layer  that  masks  the  heterogeneity  of 
the  underlying  networks,  hardware,  operating  systems  and  programming  languages 
(Britton  et  ah,  2004;  Tannenbaum  et  ah,  2002).  Another  key  element  of  conceptual 
interoperability  is  the  nature  of  each  individual  model’s  assumptions  (Garlan  et  ah, 

1995).  Both  DIS  and  HEA  presume  that  all  participants  are  trusted  agents,  in  that  they  do 
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not  deliberately  “cheat”  or  misrepresent  DVE  state.  However  if  one  system  considers 
clouds  in  visibility  determination  and  another  does  not,  then  an  unintended  “cheat”  has 
just  been  introduced.  Simplistic  solutions  to  these  challenges  are  likely  to  fail. 

The  Extensible  Modeling  and  Simulation  Eramework  (XMSE)  and  the  Eevels  of 
Conceptual  Interoperability  Model  (EICM)  are  efforts  directed  towards  addressing  both 
semantic  interoperability  and  conceptual  interoperability  (Brutzman  et  ah,  2002;  Tolk  & 
Muguira,  2003).  Eurther  discussion  on  conceptual  interoperability  is  beyond  the  scope  of 
this  thesis,  except  to  show  that  alternative  frameworks  for  distributed  virtual 
environments  supporting  military  applications  are  important  topics  for  continued 
consideration.  The  application  of  standards  and  middleware  for  enabling  interoperability 
is  relevant  to  this  thesis.  In  the  end,  what  the  “cooperative  execution  framework”  of  two 
simulations  might  be  called  is  unimportant  when  compared  to  whether  or  not  useful 
training  can  be  provided  by  the  connected  systems  (Page  et  ah,  2004). 

3.  Distributed  Interactive  Simulation  (DIS):  A  Protocol  Approach 

The  Distributed  Interactive  Simulation  (DIS)  protocol,  IEEE  1278.1a  1998,  is  an 
open-standard  protocol  that  defines  message  format  and  content  to  enable  the  connection 
of  simulations.  It  was  developed  specifically  for  real-time  applications  and  this  is 
reflected  by  the  specifications  of  the  standard.  DIS  benefits  from  being  relatively 
straightforward  conceptually  and  structurally,  but  it  has  some  limitations,  which  are 
imposed  by  its  real-time  nature. 

DIS  was  closely  patterned  after  the  distributed  interaction  protocol  for  SIMNET 
that  was  developed  in  the  1980s.  SIMNET  itself  was  developed  to  enable  distributed 
interaction  for  large  numbers  of  combatant  entities.  The  original  concept  was  eventually 
constrained  to  ground  vehicles  because  the  message-exchange  requirements  were  more 
manageable  (Miller  &  Thorpe,  1995).  The  success  of  SIMNET  led  to  the  funding  of  an 
effort  to  develop  a  standard  for  real-time  platform-level  wargaming  across  distributed 
hosts  beyond  SIMNET.  A  series  of  workshops  hosted  at  the  University  of  Central  Elorida 
led  to  the  initial  adoption  of  the  DIS  standard,  which  then  became  a  formal  IEEE  standard 
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in  1993.  The  mission  of  DIS  is  to  create  synthetic  virtual  representations  of  warfare 
environments  by  systematically  connecting  separate  subcomponents  of  simulations, 
which  reside  at  distributed  multiple  locations  (SISO,  2007). 

DIS  messages  are  called  Protocol  Data  Units  (PDUs)  and  they  are  the  means  of 
information  exchange  between  simulations.  There  are  67  standard  PDUs  in  the  current 
version  of  the  standard  (IEEE  1278).  Non-standard  PDUs  that  are  application  specific  are 
sometimes  used  as  well.  There  are  four  types  of  PDUs,  which  generally  define  the 
primary  interactions  between  participants  in  a  DIS-based  DVE.  These  are  the  Entity 
State,  Eire  (as  in  weapon  fire).  Collision,  and  Detonation  PDUs  (IEEE  1278). 

The  Entity  State  PDU  (ESPDU)  is  a  fairly  complete  representation  of  the  dynamic 
physical  state  of  the  entity  (Table  1).  It  is  an  inherently  platform-level  representation  of 
position  and  orientation  based  on  rigid-body  physics,  capturing  state  for  kinematic 
(velocity-based)  or  dynamic  (acceleration-based)  motion.  The  ESPDU  contains  elements 
such  as  location,  linear  velocity  and  others.  At  a  minimum,  an  ESPDU  will  be  576  bits 
(72  bytes)  in  length.  A  typical  ESPDU  is  144  bytes  long.  ESPDUs  generally  make  up 
more  than  70%  of  the  message  traffic  in  DIS  based  DVEs  (SISO,  2007). 

DIS  requires  several  key  assumptions  in  order  to  work  that  are  part  of  the  formal 
standard. 

•  Participating  simulation  nodes  in  a  DIS-based  DVE  are  autonomous  and 
authoritative  with  respect  to  their  own  state,  and  transmit  the  ground-truth 
of  that  state  to  other  nodes. 

•  Each  simulation  node  is  responsible  for  periodically  issuing  the  PDUs  that 
reflect  that  node’s  state. 

•  There  is  no  central  computer  with  an  authoritative  record  of  the  state  of 
that  node.  Each  simulation  node  determines  what  is  true  for  that  node 
including  the  effect  of  actions  by  other  nodes  (IEEE  1278). 

A  specific  example  considering  two  simulation  nodes  (A  and  B)  and  the  handling 
of  weapons  fire  is  descriptive.  If  simulation  node  A  fires  a  weapon  at  node  B,  node  A 
sends  a  Eire  PDU,  via  broadcast  or  multicast  transport.  The  simulation  modeling  the 
munitions  fired  then  sends  a  Detonation  PDU,  which  includes  the  impact  location  if 
applicable.  This  is  typically  the  firing  node  but  the  DIS  standard  does  not  explicitly 
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require  this  to  be  the  ease.  Upon  reeeipt  of  a  Detonation  PDU,  simulation  node  B 
determines  if  any  damage  oeeurred  as  result  of  the  detonation.  It  is  apparent  that  DIS 
relies  inherently  upon  eompletely  trustworthy  elients.  Another  key  assumption  in  this 
methodology  is  the  use  of  dead  reekoning  for  updating  physieal  loeation  and  orientation 
between  updates,  in  order  to  reduee  the  number  of  messages  that  need  to  be  transmitted 
and  to  provide  a  shared  eonsistent  state  at  all  nodes. 

While  DIS  has  been  relatively  sueeessful,  it  has  some  well-known  limitations  that 
are  a  result  of  the  faet  that  it  was  derived  from  the  SIMNET  protoeol.  SIMNET  was 
speeifieally  designed  for  interaetion  involving  dozens  to  hundreds  of  distributed  real-time 
platform  level  simulations  (Miller  &  Thorpe,  1995).  One  eonsequenee  of  the  autonomous 
node  arehiteeture  is  that  many  “heartbeat”  state -update  messages  must  be  sent,  even 
when  there  are  in  faet  no  ehanges  to  the  state  of  the  entities  at  that  node.  Even  when  some 
aspeet  of  state  does  ehange,  mueh  of  the  information  eontained  within  the  PDUs 
themselves  may  be  the  same.  The  result  of  this  is  that  while  DoD  goals  foresaw 
networked  simulations  involving  100,000  or  more  partieipants,  in  praetiee  the  highest 
numbers  of  simultaneous  partieipants  using  DIS  only  is  on  the  order  of  two  to  three 
thousand  sueh  as  was  seen  during  the  Synthetie  Theater  of  War  (STOW)  series  of 
experiments  in  the  mid  1990s  (OTA,  1995;  Keune  &  Coppoek,  1995). 
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Table  1.  The  first  144  bits  (18  bytes)  of  an  ESPDU  (From:  IEEE  1278) 


Field 

size 

(bits) 

Entity  State  Update  PDU  fields 

Proloeol  Version — 8-bit  eiuiiiieralion 

Exereise  ID — 8-bit  unsigned  integer 

96 

PDU  Header 

PDU  type — 8-bit  enumeration 

Protoeol  Family — 8-bit  enumeration 

rimestanijr — .t2-bit  unsigned  integer 

Length — 1 6-bit  unsigned  integer 

Padding — 16  bits  unused 

Site — 16-bit  unsigned  integer 

48 

Ilntity  11) 

Appliealion — 16-bit  unsigned  integer 

Entity — 16-bit  unsigned  integer 

Signitioant  researeh  has  been  conducted  to  expand  the  scalability  of  DIS-based 
DVEs,  including  an  area  of  interest  management  (AOIM)  schemes,  which  seek  to  only 
transmit  state  update  messages  to  those  nodes  that  have  a  need  for  the  information,  and 
hybrid  architectures  where  related  messages  are  aggregated  and  then  sent  collectively  to 
disaggregation  nodes  (Macedonia,  1995).  In  this  regard,  DIS  does  not  meet  the 
requirement  of  a  scalable  system  because  the  architecture  must  be  changed  in  order  to 
increase  the  size  of  the  system.  For  small-to-medium  sized  platform-level  real-time 
distributed  virtual  environments,  DIS  is  completely  suitable.  Despite  the  fact  that  its 
conceptual  framework  is  based  upon  a  twenty-year-old  protocol,  it  is  still  in  widespread 
use  in  military  DVEs  (Strassburger  et  ah,  2008;  Steel,  2000). 

4.  The  High  Level  Architecture  (HLA)  IEEE  1516 

The  High  Eevel  Architecture  (HEA)  IEEE  1516  is  a  software  architecture  that 
provides  the  ability  to  link  different  kinds  of  simulations,  simulators,  models  and  other 
tools.  Although  it  is  generally  associated  with  distributed  simulation,  the  primary 
motivation  for  its  development  was  to  support  “composable”  simulation  (Dahmann, 


21 


1999;  IEEE  1516,  2000).  Although  HEA  is  potentially  a  powerful  arehiteeture,  the 
proeess  by  whieh  it  was  developed  and  the  requirements  it  must  meet  ereated  a  strueture 
that  has  a  steep  learning  curve  for  users.  Hopefully,  an  emergent  need  for  complex 
reusable  simulations  beyond  defense  applications  might  someday  reduce  this  learning 
curve. 

Traditional  simulation  models  often  lack  two  desirable  properties:  reusability  and 
interoperability  (Kuhl  et  al.,  1999).  In  this  context,  reusability  means  that  old  simulation 
models  can  be  reused  in  different  simulation  scenarios  and  applications.  Interoperability 
means  that  the  reusable  component  simulations  can  be  combined  with  other  models  and 
simulations  without  the  need  for  recoding.  Several  HEA  design  goals  were  established  in 
response  to  these  common  needs.  They  may  be  divided  into  two  sets. 

•  The  need  to  save  existing  simulations 

•  To  maximize  the  reusability  of  the  existing  simulations  models 

•  To  make  it  possible  for  individual  simulation  models  to  be 
combined  in  order  to  model  more  complex  systems 

•  To  allow  the  individual  simulation  models  to  interact  in  a  manner 
that  supports  distributed  simulation  technologies 

•  The  need  to  allow  for  future  capabilities. 

•  To  provide  a  larger  capability  for  modeling  command  and  control 
(C2)  structures 

•  To  provide  a  flexible  framework  that  will  permit  new  technologies 
to  be  incorporated  in  the  simulation  models 

•  To  provide  simulation  models  that  can  be  immediately 
incorporated  into  developing  planning  and  control  technologies 
(Kuhl  etak,  1999) 

To  encourage  widespread  acceptance  of  HEA,  DOD  submihed  the  HEA  standards 
to  two  standards  bodies,  the  Object  Management  Group  (OMG)  and  the  IEEE.  The 
Interface  Specification  was  adopted  by  the  OMG  in  1998.  The  IEEE  adopted  the  HEA 
Standard  in  2000  as  the  IEEE  1516  series.  In  November  2000,  all  of  the  services  signed  a 
Memorandum  of  Agreement  that  mandated  HEA  as  the  standard  technical  architecture 
for  interoperability  among  DoD  simulations  (DoD  A&T,  2000). 
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a.  Features  of  HLA 


The  baseline  definition  of  the  HLA  ineludes:  HLA  Rules,  HLA  Interfaee 
Specification,  and  HLA  Object  Model  Template.  It  is  important  to  note  that  HLA  is  not 
software  per-se.  Rather  HLA  is  a  set  of  rules.  The  rules  are  divided  into  two  parts:  the 
federation  rules  and  the  federate  rules.  Federations  work  with  the  Federation  Object 
Model  (FOM)  to  interoperate.  The  federate  rules  direct  that  each  federate  must  document 
their  public  information  as  well  as  what  they  must  import  and  export  to  other  federates,  in 
order  to  comply  with  the  federation  Run-Time  Infrastructure  (RTI).  The  RTI  is  the  actual 
implementation  software  and  hardware  that  links  federates  (Figure  4).  The  RTI  manages 
six  critical  functions;  Federation  management.  Object  management.  Time  management. 
Declaration  management.  Ownership  management,  and  Data  Distribution  management 
(Kuhl  et  ah,  1999).  For  two  HLA-compliant  applications  to  interoperate,  they  must  utilize 
the  same  FOM  and  the  same  RTI  (Ryan  &  Zalcman,  2003).  Some  form  of  software 
commonality  is  also  needed  for  message  exchange.  More  detailed  technical  discussion  of 
HLA  is  beyond  the  scope  of  this  thesis. 

The  DoD-sponsored  development  of  HLA  was  given  impetus  due  to  the 
belief  that  composability  (reusability  and  interoperability)  leads  to  better  simulations  at 
reduced  cost.  Based  upon  the  historical  trend  towards  increasingly  complex  and 
expensive  “monolithic”  simulations,  this  notion  certainly  seems  reasonable.  Typically 
technology  goes  through  a  lifecycle  from  birth  to  retirement.  Initially,  it  may  be 
completely  within  the  research  and  development  (R&D)  domain,  and  then  through  the 
process  of  transfer  and  diffusion  a  new  technology  transitions  to  a  point  where  new 
products  are  developed,  ultimately  to  a  phase  where  it  undergoes  general  adoption. 
Usually  it  does  not  reach  the  phase  of  general  adoption  until  tools  and  methods  for  its 
implementation  are  relatively  stable  and  mature.  This  maturation  process  takes  time  and 
significant  effort  (Fowler  et  ah,  1993).  Due  to  the  apparent  urgency,  initially  DoD  tried  to 
supply  all  of  the  elements  of  the  software  economy  that  normally  accompany  the 
development  of  a  new  technology.  This  includes  the  software  producers,  consumers, 
publicity,  training,  support,  testing,  and  perception  of  economic  benefit  (Kuhl  et  ah, 

1999). 
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Support  Utilities 


i 

- 

Constructive 

Simulation 


Communicatons  Interfaces/Gateways 


Runtime  Infrastructure: 

All  interactions  between  federates  must  occur  via  the  RTI 


Figure  4.  Conceptual  Overview  of  HLA. 


Generally,  a  Federation  Object  Model  is  described  as  what  the  various 
federates  publish  and  subscribe  to.  This  is  a  deceptively  simple  description.  It  follows 
that  in  order  to  build  a  federation  object  model,  the  developer  must  specify  all  of  the 
publish-and-subscribe  characteristics  for  every  federate.  Except  for  very  simple 
federations  this  must  be  carefully  defined  beforehand.  This  requires  a  software 
engineering  approach  and  a  capacity  for  high-level  thinking,  and  discourages  rapid 
changes  at  run-time.  Anecdotally,  for  large-scale  simulations,  federation  development 
time  is  measured  in  many  weeks  or  months  (Ceranowicz  et  ah,  2002;  Kim  et  ah,  2005). 

b.  Limitations  of  HLA 

Non-military  application  of  HLA  has  primarily  been  limited  to  the 
research  community  with  a  few  exceptions.  The  general  sentiment  is  that  HLA  offers  too 
much  irrelevant  capability  for  industrial  or  commercial  applications.  The  perception  of 
commercial  simulation  developers  is  that  there  is  no  economic  benefit  to  themselves  from 


24 


reusability  and  interoperability.  These  faetors,  eoupled  with  the  steep  learning  eurve 
assoeiated  with  eompetenee  (let  alone  expertise)  in  HLA,  have  made  its  non  defense- 
seetor  reeeption  lukewarm  at  best  (Boer  et  ah,  2003). 

HLA  was  designed  to  support  eomposable  military  simulations.  HLA 
eomplexity  is  perhaps  “as  simple  as  possible  but  no  simpler”  (Kuhl  et  ah,  1999)  but 
nevertheless  faees  barriers  to  widespread  adoption  (Strassburger  et  ah,  2008).  A  reeurring 
theme  of  diseussion  is  the  desire  for  the  subsetting  of  major  HLA  funetionality  to  enable 
widespread  adoption.  Other  arehiteetures  are  eapable  of  providing  analogous  if  redueed 
funetionality  sets  with  mueh  less  eomplexity. 

A  fundamental  limitation  is  that  the  HLA  speeifieation  does  not  require 
HLA  RTI  implementations  to  interoperate  with  eaeh  other  and  speeifies  no  networked 
data  formats  for  interoperability  between  different  HLA  implementations  (IEEE  1516, 
2000).  This  has  the  result  of  leaving  otherwise  HEA-eompliant  simulations  unable  to 
eonneet  with  eaeh  other.  Sponsors  of  HEA-eompliant  simulations  thus  potentially 
beeome  “loeked  in”  by  eommereial  lieenses  on  speeifie  RTIs. 

5.  Generic  Massively  Multiplayer  Online  Game  (MMOG)  Architectural 

Description 

Massively  Multiplayer  Online  Games  (MMOGs)  are  a  subset  of  distributed  virtual 
environments  (DVEs).  While  MMOGs  are  obviously  games  first  and  foremost,  they  faee 
information  exehange  requirements  similar  to  DVEs.  Strip  away  the  game  faeade  (Eigure 
5)  and  they  are  more  like  military  DVEs  than  not.  Some  general  prineiples  of  distributed 
virtual  environments  ean  be  seen  in  the  following  examination  of  the  eomponents  and 
eonneetions  of  a  MMOG  system.  This  examination  is  eondueted  within  the  eontext  of 
using  MMOG  arehiteeture  to  enable  intereonneetion  and  interoperability  of  militarily 
useful  virtual  environments  and  simulations. 
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Figure  5.  Screenshots  of  two  popular  MMOGs.  (left)  Lineage®  (NCsoft),  (right) 
Navy  Field®  (SD  Entemet). 


a.  Qualitative  Definition  of  the  Term  Massive  Multiplayer  Online 
Game 

A  massively  multiplayer  online  game  is  one  in  which  a  large  number  of 
people  can  interact  simultaneously  over  a  network.  They  are  able  to  join  and  leave  at  any 
time  and  they  can  expect  that  their  own  game  state  will  not  change  while  they  are  offline. 
The  game  environment  is  “persistent.”  The  term  “large  number”  and  the  length  of  time, 
which  makes  a  game  “persistent,”  are  application  dependent  and  not  rigorously  defined. 
Typically,  most  commercial  MMOGs  will  last  several  years,  and  some  have  been 
operating  continuously  for  almost  a  decade  (Hall  &  Novak,  2008).  Contrast  this  with  the 
typical  military  distributed  virtual  environment  whose  persistence  is  usually  measured  in 
hours  or  days  at  most.  With  respect  to  the  “massive”  qualifier,  any  hard  number  selected 
is  often  completely  arbitrary.  Multiuser  virtual  environments  of  any  size  have  the  same 
fundamental  functional  requirements.  It  is  also  important  to  note  that  while  many 
MMOGs  may  have  hundreds  of  thousands  of  subscribers,  and  some  have  seen  more  than 
50,000  users  online  simultaneously,  none  allow  more  than  a  few  thousand  participants  to 
simultaneously  interact  with  one  another.  This  is  comparable  to  military  DVEs  such  as 
the  Synthetic  Theater  of  War  (STOW)  experiments  (Eeng  et  ah,  2007;  IGDA,  2004; 
Keune  &  Coppock,  1995). 
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MMOG  developers  are  not  free  from  the  consequenees  of  the  eonsisteney- 
throughput  tradeoff.  Many  of  the  load-management  schemes  employed  in  the 
development  of  large  experimental  military  distributed  virtual  environments  such  as  area- 
of-interest  management  (AOIM)  and  hybrid  architectures  are  also  employed  by 
commercial  MMOGs  (Bogojevic  &  Kazemzadeh,  2003;  Macedonia,  1995;  Lecky 
Thompson,  2009).  While  other  military  synthetic  exercises  such  as  Millennium  Challenge 
2002  have  included  as  many  at  30,000  participants,  they  were  not  interacting  directly 
with  one  another.  Separate  federations  of  participants  were  used  to  feed  command  and 
control  systems  primarily  for  visualization  (Ceranowicz  et  ah,  2002).  The  qualifying 
characteristics  of  an  MMOG  are  therefore  persistent  state  and  inherent  scalability  (IGDA, 
2004).  Scalability  explicitly  means  that  the  system  can  handle  larger  and  larger  numbers 
of  participants  gracefully  or  without  more-than-minor  changes  to  the  system  architecture 
(Diehl,  2001;  Singhal  &  Zyda,  1999). 

b.  MMOGs  Are  a  Game  Service 

MMOGs  are  games.  This  seemingly  obvious  statement  serves  to 
emphasize  that  their  design  and  system  architecture  are  driven  by  this  reality.  Briefly,  a 
game  can  be  described  as  a  rule-based  system  in  which  the  player(s)  must  overcome 
some  obstacle  or  challenge  to  reach  the  predetermined  end-state  of  the  game.  Game 
actions  are  typically  constrained  by  well-defined  rules  and  a  story  or  narrative  that  sets 
the  abstract  context  for  the  challenges,  the  rules,  and  the  end  state  (Narayanasamy,  2006; 
Bindley,  2003).  Commercial  games  are  designed  largely  to  maximize  sales,  repeated 
usage,  and  profits.  MMOGs  primarily  differ  from  stand-alone  games  in  that  they  are 
inherently  more  of  a  service  than  a  product.  Figure  6  depicts  a  generalized  MMOG 
service  architecture.  It  follows  that  a  major  part  of  any  MMOG  system  is  customer 
management.  While  a  complete  implementation  must  include  a  significant  customer- 
management  component,  only  the  game-services  component  is  directly  relevant  to  this 
thesis.  Game  services  allow  players  to  interact  with  other  participants  and  enable  player 
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actions  to  change  the  game  state.  Since  it  is  diffieult  to  enforce  a  consistent  game 
experience  among  different  system  arehiteetures,  almost  all  commereial  MMOGs  today 
are  client-server  systems  (Hall  &  Novak,  2008;  IGDA,  2004). 


Figure  6.  Abstract  architecture  for  massively  multiplayer  online  game  (From;  Hall  & 
Novak,  2008). 


Game  services  are  described  sehematieally  in  Figure  7.  They  are  summarized  as  follows. 

•  In  addition  to  managing  the  user  interface,  the  elient  process  the  users 
input  in  the  form  of  state  update  messages  and  transmits  them  to  the 
server. 

•  The  server  then  calculates  a  new  game  state  by  applying  the  reeeived  user 
actions  and  the  game  logic  to  the  current  game  state.  As  a  result  of  this 
calculation,  the  state  of  several  dynamic  entities  has  changed. 

•  Finally,  the  new  game  state  is  transferred  back  to  the  clients  (Bogojevic  & 
Kazemzadeh,  2003). 
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Figure  7.  The  fundamental  game  serviees  of  a  client-server  based  distributed  virtual 
environment  (From;  Glinka  et  al.,  2007). 


c.  MMOGs  and  Scalability 

Commercial  MMOGs  almost  all  have  client-server  architecture  for  reasons 
outlined  earlier.  However,  it  is  obvious  that  a  single  server  cannot  handle  an  infinite 
number  of  client  connections;  CPU  cycles  on  the  server  will  eventually  run  out,  or  the 
server  will  be  unable  to  handle  a  large  number  of  socket  connections  or  bandwidth 
availability  becomes  saturated.  These  problems  lead  to  a  scale-out  solution,  in  which  a 
cluster  of  computers  on  the  server  side  each  handles  separate  client  connections  and  game 
state.  This  can  be  done  in  a  variety  of  ways.  One  of  the  most  popular  is  “sharding”  in 
which  parallel  instances  of  the  game  state  are  each  handled  by  a  single  server  node.  One 
of  the  disadvantages  of  sharding  is  that  participants  on  separate  shards  cannot  interact 
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with  one  another  (Ye  &  Cheng,  2006;  Emilsson  et  ah,  2009).  Zoning  is  another  load 
distribution  seheme  in  whieh  partieipants  are  assigned  to  server  nodes  based  upon  the 
participant’s  virtual  location  in  the  game  environment.  The  overall  game-play  area  is 
divided  up  into  multiple  geographic  regions,  and  players  are  assigned  to  a  server 
dedicated  to  each  virtual  region.  However,  this  approach  can  have  its  own  problems, 
including  low  server  utilization.  A  server  must  be  associated  with  a  region  even  if  few 
people  are  there  at  the  time,  which  can  lead  to  over-provisioning  the  data  center 
(McGregor  et  ah,  2009;  Nae  et  ah,  2008).  Often,  zoning  and  sharding  are  combined  with 
popular  game  regions  being  duplicated  over  several  server  nodes  (Glinka  et  ah,  2007;  Lu 
et  ah,  2006;  Ye  &  Cheng,  2006). 

There  are  a  few  alternatives  such  as  the  single-shard  architecture  used  for 
the  space-based  MMO  Eve  Online.  Eve  Online  applies  a  design  philosophy  that  holds 
that  the  game  participants  are  the  primary  content.  The  developers  take  advantage  of  the 
fact  that  the  distance  scales  in  the  game  environment  create  natural  aggregation  points  at 
certain  game  locations.  In  this  single-shard  architecture,  there  is  only  one  instance  of  the 
game  state  and  any  player  from  the  global  base  may  interact  with  any  other  player 
provided  they  are  at  the  same  game  location.  In  the  case  of  Eve  Online,  an  exceptionally 
large  cluster  is  used  to  achieve  this  scale  (Emilsson  et  ah,  2009). 

Despite  the  differing  motivations  for  their  development,  militarily  useful 
distributed  virtual  environments  and  MMOGs  faces  similar  challenges.  The  next  section 
examines  the  similarity  of  the  information-exchange  requirements  between  typical 
MMOGs  and  military  DVEs. 

6,  Comparison  of  MMOG  Network  Traffic  to  Military  DVE  Traffic 

The  consistency-throughput  tradeoff  means  that  developers  of  DVEs  cannot 
arbitrarily  decide  upon  the  rate  at  which  information  flows  between  the  participants.  Eor 
commercial  MMOG  developers,  who  desire  to  handle  large  numbers  of  simultaneous 
players  while  maintaining  a  consistent  game  experience,  this  has  a  direct  economic 
impact.  A  look  at  the  network  traffic  patterns  of  various  DVEs  shows  that  with  regard  to 
network  traffic,  MMOGs  and  some  large  scale  DVEs  are  analogous.  While  high  update 
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rates  may  be  required  for  small-scale  fast-reflex  DVEs  such  as  first-person  shooter  (FPS) 
type  games  or  air  combat  maneuvering,  it  is  often  true  that  for  loosely  coupled  interaction 
relatively  modest  update  rates  are  acceptable  (Bonk  &  Dennen,  2005). 

Even  during  the  development  of  early  DVEs  such  as  SIMNET  and  NPSNET,  the 
importance  of  measuring  network  traffic  was  realized  (Macedonia,  1995).  Due  to  the 
impact  that  game  traffic  has  on  networks  and  the  cost  of  maintaining  sufficient  capacity 
to  handle  large  numbers  of  simultaneous  participants,  the  body  of  research  on  game 
traffic  analysis  and  quality  of  service  requirements  (QoS)  for  games  has  grown  in  parallel 
with  the  growth  in  the  number  of  MMOG  subscribers.  This  type  of  research  can  be 
broadly  categorized  into  two  categories.  The  first  is  the  experimental  determination  of  the 
effects  of  latency  and  throughput  on  the  D  VE  user  experience,  and  the  second  is  the 
characterization  of  the  network  traffic  of  existing  DVEs  (Kwok,  2006). 

a.  Tightly  and  Loosely  Coupled  DVEs 

The  notion  of  coupling  is  important.  This  term  is  borrowed  from  systems 
engineering  and  distributed  computing  and  refers  to  the  degree  that  the  behavior  or 
processes  of  a  node  depends  on  knowledge  about  the  behavior  of  another  node.  In  a 
DVE,  entities  are  either  closely  or  loosely  coupled.  An  example  of  tight  coupling  might 
be  a  group  of  tanks  in  a  closely  spaced  formation  moving  at  high  speed.  In  order  to 
maintain  safe  spacing  and  fidelity,  and  to  avoid  interfering  with  mutual  fields  of  fire,  each 
entity  needs  to  have  sufficiently  accurate  and  timely  information  about  the  other  entities 
in  the  formation  (Chassot  et  ah,  1999;  Rushby,  1994).  In  general,  tightly  coupled  systems 
do  not  function  well  when  message  delays  exceed  100  ms.  In  contrast,  an  example  of 
loosely  coupled  entities  might  be  two  tanks  20  kilometers  apart.  The  notion  of  coupling  is 
important  because  network  QoS  requirements  are  generally  determined  by  whether 
entities  are  tightly  or  loosely  coupled.  Note,  however,  there  is  no  hard  line  dividing 
tightly  and  loosely  coupled  systems.  A  qualitative  distinction  is  typically  decided  by  the 
interaction  needs  of  the  application  models  (Kuiper  &  Eemmers,  2000;  Singhal  &  Zyda, 
1999). 
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b.  Delay  and  Throughput  for  Typical  Military  DVEs 


The  acceptable  delay  and  throughput  for  a  typical  military  DVE  needs  to 
be  considered  for  any  effective  application.  Unfortunately,  most  of  the  literature  on 
acceptable  QoS  for  military  DVEs  is  experiential  in  nature.  Nevertheless,  experience  has 
shown  that  delays  of  less  than  or  equal  to  100  milliseconds  (ms)  are  acceptable  for  tightly 
coupled  real-time  platform-level  DVEs  (Chassot  et  ah,  1999;  DIS  Steering  Committee, 
1994;  Singhal  &  Zyda,  1999).  Throughput  for  tightly  coupled  real-time  platform  level 
DVEs  depends  upon  the  information-exchange  requirements  for  the  type  of  platform  that 
an  entity  represents.  By  utilizing  traffic  management  schemes  such  as  dead  reckoning,  the 
acceptable  throughput  ranges  from  one  update  per  second  for  ground  vehicles  to  30 
updates  per  second  for  articulated  human  entities  (Singhal  &  Zyda,  1999;  Macedonia, 
1995). 

Eor  loosely  coupled  platform  level  military  DVEs,  delays  of  up  to  300  ms 
are  acceptable.  Acceptable  throughput  rates  again  depend  upon  the  type  of  entity  being 
simulated.  Unfortunately,  the  limited  amount  of  research  on  acceptable  QoS  for  military 
specific  DVEs  does  not  differentiate  between  throughput  requirements  for  tightly  or 
loosely  coupled  entities.  Experience  from  the  STOW  experiments  indicates  that  one  to 
three  updates  per  second  are  acceptable  in  this  case  (Keune  &  Coppock,  1995).  Analysis 
that  only  reports  average  update  rates  should  be  used  with  caution,  as  this  metric  typically 
includes  periods  of  time  in  which  entities  were  stationary  (Macedonia,  1995;  Cheung  & 
Eoper,  1994). 

The  above  examples  are  typical.  An  extreme  case  of  a  tightly  coupled 
system  includes  remote  surgery  applications  with  haptic  feedback.  Minimum  acceptable 
throughput  for  this  type  of  application  is  on  the  order  of  1000  updates  per  second  (Yap  et 
ah,  2005;  Choi  et  ah,  2004).  At  the  other  extreme,  computer-assisted  exercises  in  which 
all  interaction  between  entities  is  highly  abstracted  can  require  relatively  infrequent 
updates  from  the  participating  entities  (Ceranowicz  et  ah,  2002). 
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c. 


Throughput  and  Delay  in  MMOGs 


There  are  several  papers  deseribing  experimental  determination  of 
aeeeptable  delay  and  throughput  for  networked  games  in  the  literature.  Not  surprisingly, 
aeeeptability  is  broadly  eategorized  by  game  genre. 

d.  Massively  Multiplayer  Online  Role-Playing  Games  (MMORPG) 

Massively  multiplayer  online  role  playing  games  (MMORPG)  are  best 
deseribed  as  loosely  eoupled  distributed  virtual  environments  even  though  they  have 
signilieant  interaetivity.  This  is  beeause  game  interaetion  is  event  based.  When  a  player 
elicks  on  a  monster  to  attack  it,  an  attack  message  is  sent.  The  combat  resolution  of 
success  or  failure  then  occurs  at  the  server.  There  is  no  parallel  processing  between  the 
client  and  the  server  to  handle  the  flight  of  an  arrow  for  example.  A  typical  MMORPG  is 
Everquest  II  (Sony,  2009).  For  these  types  of  games,  investigators  have  reported 
degraded  user  experience  at  delays  of  approximately  200  ms  (Ries  et  ah,  2008;  Chen  et 
ah,  2006).  In  some  cases,  however,  delays  on  the  order  of  1000  ms  have  been 
demonstrated  to  have  almost  no  effect  on  self-reported  user  experience  (Fritsch  et  ah, 
2005).  Within  this  genre,  update  rates  range  from  three  per  second  for  stationary  entities 
to  seven  per  second  for  entities  involved  in  game  combat  (Park  et  ah,  2005;  Chen  et  ah, 
2005). 


e.  Massively  Multiplayer  Real-Time  Strategy  Games  (MMRTS) 

Massively  multiplayer  real-time  strategy  games  (MMRTS)  are  in  relative 
infancy  compared  to  MMORPGs.  Typically,  the  player  does  not  closely  control  a  single 
avatar  but  exercises  general  direction  and  oversight  over  a  large  number  of  entities. 
Networked  real-time  strategy  games  (RTS)  are  quite  mature  but  these  are  limited  to  a  few 
dozen  players.  Warcraft  III  is  a  typical  example.  For  these  types  of  games,  user 
experience  often  degraded  once  interaction  delays  exceeded  800  ms.  In  at  least  one  case, 
user  effectiveness  at  a  set  of  specified  in-game  tasks  remained  unaffected  with  update 
delays  of  up  to  2000  ms.  The  minimum  rate  for  real-time  strategy  games  is  about  four 
updates  per  second,  with  six  updates  per  second  being  typical.  Peaks  of  up  to  15  updates 
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per  second  are  seen  during  combat  sequences  involving  large  numbers  of  entities  at  each 
node  (Claypool,  2005;  Bettner  &  Terrano,  2001).  Unfortunately,  the  literature  that  covers 
this  genre  is  not  yet  directly  relevant  to  the  challenges  of  MMRTS  because  only 
conventional  (non-massive)  real-time  strategy  games  have  been  studied  and  reported. 
Developers  of  conventional  RTS  games  are  able  to  use  schemes  such  as  lock-step 
simulation  processing.  Roughly,  instead  of  passing  the  status  of  each  unit  in  the  game,  the 
exact  same  simulation  is  run  on  each  machine  with  only  commands  being  distributed  to 
the  other  participants  (Bettner  &  Terrano,  2001).  This  is  only  practicable  for  session- 
based  games  where  all  participants  begin  with  the  exact  same  state. 

f.  Platform-level  Simulation  Games 

Platform-level  simulation  games  include  both  vehicle  simulators  and  so- 
called  first-person  shooters  (FPS).  For  vehicle-based  simulations  such  as  driving 
simulation  games,  user  experience  and  performance  are  relatively  unaffected  by  delays  of 
up  to  100  ms.  Beyond  this  point,  however,  performance  degrades  rapidly.  Lap  times 
around  a  simulated  track  in  one  study  were  100%  worse  when  user- interaction  delay  went 
from  150  ms  to  250  ms.  For  car-racing  games,  12  updates  per  second  80  ms  delay)  is  a 
typical  throughput  rate.  Less-dynamic  platform-level  simulations  do  not  require  as  high 
an  update  rate  (Pantel  &  Wolf,  2002;  Yasui  et  ah,  2005).  For  FPS  games,  in  at  least  one 
case,  no  degradation  to  player  effectiveness  at  game  tasks  is  experienced  with  delays  up 
to  100  ms.  Player  effectiveness  degrades  rapidly  for  precise  tasks  such  as  precision 
shooting  above  100  ms  latency.  Networked  FPS  games  have  update  rates  of  from  20  to 
40  updates  per  second  (Beigbeder  et  ah,  2004).  These  demanding  throughput 
requirements  make  implementation  of  a  scalable  FPS  problematic.  Planetside®,  a  science 
fiction  combat-oriented  MMOG  in  which  hundreds  of  players  can  participate  in  a  single 
fight,  is  a  notable  example  of  a  FPS  MMOG  (Sony,  2009). 

g.  The  Effect  of  Architecture  on  Bandwidth  Requirements 

The  state -update  behavior  per  node  for  military  DVEs  and  MMOGs  has 
long  been  considered.  Bandwidth  requirements  are  dependent  upon  per-node  behavior. 
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the  information-exchange  requirements  of  the  participants  in  the  DVE,  and  the  network 
architecture  (Pelligrino  &  Dovrolis,  2003).  Since  game  developers  control  the 
implementation  of  all  nodes,  the  information  exchange  requirements  for  games  are 
significantly  smaller  than  that  for  militarily  useful  DVEs.  Game  nodes  usually  know 
exactly  the  type  and  behavior  of  all  possible  entities  at  every  other  game  node.  Eor 
online  games,  small  packet  size  is  used  in  order  to  reduce  bandwidth  and  latency  (Chen  et 
ah,  2005;  Eeng  et  ah,  2005).  A  typical  update  message  from  the  client  to  the  server  in  a 
MMORPG  is  30-60  bytes  long  following  the  protocol  header.  Compare  this  small  size  to 
a  typical  DIS  ESPDU  at  144  bytes.  At  six  updates  per  second,  the  bandwidth  impact  is  on 
the  order  of  two  Kbps/sec.  Since  servers  must  send  partial  or  complete  updates  of 
complete  relevant  game  state  to  clients  in  MMOGs,  server-to-client  messages  vary 
widely  in  size.  Some  comparisons  between  HEA  and  DIS  have  shown  that  their 
bandwidth  requirements  are  similar  (USAE  ASC,  1998). 

These  considerations  naturally  brings  up  the  subject  of  DVE  architecture. 
As  discussed,  MMOGs  almost  exclusively  use  client-server  architecture  in  order  to 
control  the  user  experience.  This  means  that  the  average  bandwidth  requirement  at  the 
server  is  a  multiple  of  the  number  of  clients  connected  to  that  server.  This  does  not 
consider  peaks  in  traffic  for  which  sufficient  capacity  must  be  available.  Even  a  dedicated 
T1  line  (1.544  Mbps)  to  the  server  will  only  accommodate  on  the  order  of  600 
simultaneous  clients  (Macedonia,  1995).  It  is  assumed  that  some  sort  of  interest 
management  occurs  at  the  server  to  prevent  overloading  the  client’s  network  capability. 
Eor  a  strict  peer-to-peer  architecture  using  DIS  in  which  there  is  no  server,  every  client 
broadcasts  its  state  update  messages  in  the  form  of  ESPDUs.  Six  hundred  such  clients 
might  theoretically  require  on  the  order  of  3  Mbps  of  dedicated  bandwidth  at  every  peer. 
Clearly,  this  is  unworkable  in  a  wide-area  network  (WAN)  environment.  In  fact,  entity 
update  rates  vary  significantly  and  in  practical  large-scale  DVEs,  hybrid  architectures  are 
used  instead  (Diehl,  2001;  Singhal  &  Zyda,  1999). 
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h.  The  Question  of  Acceptability 

Many  MMOGs  are  able  to  maintain  high  levels  of  interactivity  at  what 
might  normally  be  considered  unacceptable  QoS.  Acceptability  is  determined  by  the 
objective  of  the  specific  DVE  being  considered.  With  current  networking  technology  it  is 
difficult  to  achieve  tightly  coupled  entity-level  simulation  in  real  time  for  numerous 
entities  in  a  wide-area  network  (WAN)  environment.  In  fact,  most  large-scale  widely 
distributed  military  DVEs  used  to  date  have  been  focused  upon  training  objectives  other 
than  platform-level  crew  proficiency.  Distributed  simulation  is  primarily  used  as  a  tool 
for  the  training  of  multiple  crews  or  staff  as  a  coordinated  military  unit.  The  major 
interest  has  been  focused  on  collective  and  not  individual  training.  Even  SIMNET,  the 
progenitor  of  platform-level  large-scale  DVEs,  had  a  design  goal  to  make  the  crews  and 
units,  not  the  device,  the  center  of  attention  in  the  simulation  (Miller  &  Thorpe,  1995). 
While  military  DVEs  have  increased  information-exchange  requirements  because  of  the 
need  to  support  interoperability,  this  is  a  difference  in  degree  and  not  type  between 
militarily  useful  DVEs  and  commercial  MMOG  architectures. 

7.  Comparing  the  DIS/HLA  Conceptual  Model  to  MMOG  Architecture 

DIS  and  to  a  lesser  extent  HLA  conform  to  the  concept  that  autonomous  nodes  are 
trustworthy  clients  and  they  are  authoritative  over  their  own  state.  Games  assume  that 
some  or  many  clients  will  cheat  and  for  this  reason  enforce  consistency  at  the  server.  In 
fact,  cheating  can  occur  without  intent  even  by  trustworthy  clients  because  of  differences 
in  combat  models.  In  effect,  clients  can  cheat  even  though  they  are  trustworthy,  due  to  the 
nature  of  unexpected  interaction  consequences  that  emerge  from  different  combat  models 
or  differences  in  terrain  representation  (Johnson  et  ah,  2004). 

Existing  military  DVEs  and  simulators  cannot  be  arbitrarily  connected  using 
MMOG  architecture  without  breaking  some  of  the  governing  standards  of  both  DIS  and 
HEA.  In  fact  a  great  deal  of  middleware  has  already  been  developed  to  enable  more 
useful  DVEs  while  maintaining  consistency  with  existing  standards.  MMOG  systems  can 
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be  employed  as  middleware  or  as  the  core  of  a  military  DVE  to  improve  their  utility  and 
ease  of  use.  The  similar  nature  of  the  use  of  the  DIS  protocol  and  architecture  makes  it 
particularly  appealing  for  adaption  to  MMOG  approaches. 

E.  APPLYING  MMOG  ARCHITECTURE  TO  MILITARY  DISTRIBUTED 
VIRTUAL  ENVIRONMENTS 

Upon  considering  this  survey  analysis,  the  first  basic  use  that  can  be  made  of  an 
MMOG  system  for  a  military  DVE  is  as  middleware  that  connects  and  mediates  between 
distributed  entities  interacting  in  a  physically  realistic  virtual  environment  and  a  scalable 
MMOG  server. 

1.  Use  Cases  for  MMOG  Middleware 

The  lowest  level  is  to  implement  a  MMOG  system  as  a  DIS-Packet  Server.  The 
motivation  for  this  implementation  includes  the  following. 

•  Abstracting  away  the  networking  component 

•  Allowing  persistent  server-side  entities  with  their  own  behavior 

•  Implementing  interest  management  schemes  on  the  server  side 

•  Development  of  a  client-level  DIS  Gateway,  which  aggregates  traffic  and 
sends  it  over  a  single  channel  to  the  server  that  then  interacts  with  another 
server  (Singhal  &  Zyda,  1999) 

The  next  level  is  to  add  game  logic  on  the  server  to  provide  the  following. 

•  Enforcement  of  consistent  state  via  standardized  damage  resolution 

•  Collision  checking 

•  Sophisticated  server  side  entities 

•  “Gap”  frame  generation  via  dead-reckoning  of  client  entities  at  the  server 
(Eujinoki,  2006) 

•  hogging  and  reporting  of  DVE  activity 
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2,  Using  an  Open-Source  MMOG  Package  to  Implement  Simulation 
Middleware 

Project  Darkstar,  to  be  examined  later,  is  a  free  and  open-source  MMOG  toolkit 
from  Sun  Microsystems.  For  the  purpose  of  this  section,  the  Project  Darkstar  middleware 
is  abstracted  as  a  mechanism  for  managing  the  distribution  and  maintenance  of  the  DVE 
state.  A  real-time  platform  level  DVE  using  DIS  is  considered. 

This  work  is  specifically  focused  upon  DIS  because,  for  real-time  platform-level 
simulations,  DIS  is  completely  suitable  and  this  type  of  simulation  encompasses  the 
majority  of  military  simulations.  DIS  traffic  can  also  be  used  to  reflect  ongoing  status 
changes  in  command  and  control  (C2)  systems.  Existing  heterogeneous  platform-level 
simulations  and  tactical  systems  might  thus  become  interconnected  using  MMOG 
architecture.  This  work  shows  that  this  type  of  architecture  can  be  implemented  without 
violating  the  rules  and  assumptions  of  DIS.  Ongoing  work  at  NPS  is  investigating 
bridging  between  DIS-style  network  protocols  with  military  C4I  data  streams  by  building 
a  track-data  conversion  hub. 

F.  SUMMARY 

DIS,  which  has  its  genesis  in  SIMNET,  was  designed  to  support  real-time 
platform  level  distributed  virtual  environment  of  at  most  a  few  hundred  entities. 
Alternative  architectures  have  made  larger  systems  possible  if  unwieldy.  HEA  defines 
mechanisms  beyond  a  platform-centric  model  to  enable  composability,  but  in  practice, 
has  not  proven  significantly  more  scalable  or  robust  than  DIS.  Evidence  of  this  is  the  fact 
that  DIS  is  still  commonly  used  for  real-time  platform  level  distributed  simulation  within 
the  Defense  sector  and  integrated  within  HEA-based  simulations.  MMOGs  are  growing 
rapidly  and  are  receiving  significant  interest  and  attention.  Although  facing  similar 
challenges  with  developers  of  MMOGs,  developers  of  militarily  useful  DVEs  must  also 
be  concerned  with  interoperability.  Nevertheless,  it  is  possible  and  advisable  to  apply 
commercial  MMOG  development  to  military  DVE  applications.  Various  tradeoffs  and 
alternative  approaches  provide  a  rich  basis  for  continued  work. 
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III.  RELATED  WORK 


A,  INTRODUCTION 

Large-scale  DVEs  and  MMOGs  are  not  useful  or  interesting  simply  because  they 
might  work  better  than  systems,  which  allow  only  a  few  participants.  Generally,  other 
advantages  dominate.  Their  strength  lies  in  bringing  physically  distributed  participants 
together  in  a  shared  environment  with  many  potentially  complex  interactions.  This  has 
many  helpful  uses  ranging  from  knowledge  solicitation  to  distributed  mission  operations. 
There  are  ongoing  efforts  to  mature  new  scalable  hybrid  architectures  as  well  as  to 
develop  middleware  that  abstracts  some  of  the  complexity  of  MMOG  implementation 
away  from  the  application  developer.  Open  standards  allow  existing  simulations  and 
models  to  be  incorporated  into  a  heterogeneous  DVE. 

B,  OFFICE  OF  NAVAL  RESEARCH  (ONR)  MASSIVE  MULTIPLAYER 

ONLINE  WARGAME  LEVERAGING  THE  INTERNET  (MMOWGLI) 

1.  Purpose 

Massive  Multiplayer  Online  Wargame  Leveraging  the  Internet  (MMOWGLI)  is 
an  Office  of  Naval  Research  (ONR)  concept-development  project  that  seeks  to  marry  the 
concepts  of  immersive  alternate  reality  games  with  massively  multiplayer  online  games, 
demonstrating  their  application  to  a  real  scenario  of  interest  to  the  Navy.  An  alternate 
reality  game  is  one  that  uses  the  real  world  as  a  platform.  Conceptually,  they  attempt  to 
combine  the  elements  of  real  life  with  an  interactive  artificial  narrative  (IGDA,  2006).  It 
is  expected  that  by  massively  scaling  up  the  participant  pool,  the  ability  might  be  gained 
to  explore  more  novel  combinations  and  complex  interactions  of  ideas,  producing 
insights  that  are  otherwise  not  forthcoming,  or  might  be  hard  to  predict  using  traditional 
methods.  Continuous  on-line  play  is  intended  to  significantly  reduce  the  fixed  overhead 
cost  of  conducting  a  scenario-driven  exercise  (Jensen,  2008). 


39 


2.  Implementation  and  Looking  Ahead 


MMOWGLI  is  not  a  technology  development  project.  The  objective  of  the  project 
is  to  utilize  existing  technologies  in  a  new  way.  The  discovery  phase  began  in  2008  with 
a  detailed  exploration  into  the  use  of  MMOGs  for  other  than  entertainment  purposes. 

The  purpose  of  MMOWGLI  is  enhanced  distributed  collaboration  specifically  for 
knowledge  solicitation.  ONR  has  developed  several  detailed  notional  scenarios  that  serve 
as  use  cases  for  the  MMOWGLI  project.  This  project  is  a  credible  use  of  MMOG 
technology  because  it  focuses  upon  social  interactions  and  not  tightly  coupled  real-time 
interaction.  While  the  focus  is  on  role-playing,  a  rich  shared  environment  might  even 
allow  the  participation  of  platform  level  entities  and  their  crews  as  stand-ins  for 
computer-controlled  units.  Many  interesting  outcomes  are  possible. 

C.  USAF  DISTRIBUTED  MISSION  OPERATIONS  ENVIRONMENT 

1.  Program  Purpose 

The  Distributed  Mission  Operations  (DM0)  Environment  is  a  persistent  training 
simulation  network  that  provides  an  on-demand  virtual/constructive  environment  for 
mission  training  of  air  crew  and  C4I  operators.  It  is  specifically  able  to  operate  in  an 
environment  over  WAN  distances,  through  firewalls,  and  constrained  by  high  latencies 
(USAF  ASC,  1998). 

2,  Current  Architecture 

DM0  uses  the  Distributed  Missions  Operations  Network  (DMON),  which 

consists  of  60  nodes,  at  over  30  locations  worldwide.  DMON  is  a  Virtual  Private 

Network  run  from  a  Network  Operations  Center  (NOC)  in  Orlando,  FL  (Figure  8).  The 

individual  endpoint  Mission  Training  Center  (MTC)  LAN  receives  all  outbound  network 

traffic  from  the  simulator  and  forwards  it  to  a  stream  manager  component  that  determines 

the  appropriate  data  stream(s)  needed  for  each  of  the  participants.  After  stream 

duplication,  a  distributor  component  then  distributes  the  stream  among  the  remote  MTC 

sites  (McVearry  et  ah,  2008).  DM0  is  expected  to  always  be  a  world  in  which  multiple 
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standards  will  overlap  and  co-exist.  Specifically,  both  HLA  and  DIS  are  used  in  a 
federation  of  simulation  systems.  There  is  no  indication  that  DMO  will  use  less  DIS  in 
the  future. 

One  of  the  primary  component  groups  of  DMO  is  the  Test  and  Training  Enabling 
(TENA)  middleware,  repository,  and  logical  range  data  archive  (Noseworthy,  2008).  This 
system  is  only  superficially  similar  to  HLA  and  requires  a  gateway  to  participate  in  an 
HLA  federation. 


•  -  Active  Hub  Centers,  Contractor  Sites  -  9  sites  (12  by  End  CY  08) 


figure  8.  Distributed  Mission  Operations  Network,  (from;  McVearry  et  ah,  2008) 


3,  Future  Possibilities 

Analysis  of  the  potential  of  DMO  anticipates  that  commercial  virtual  environment 

technology  will  leapfrog  the  DoD  because  of  the  tens  of  thousands  of  developers  working 

to  advance  the  commercial  field.  It  is  envisioned  that  a  hypothetical  fusion  of  World-of- 

Warcraft  like  technology  and  a  Second  Life  type  system  all  built  upon  real-world 

databases  that  accurately  model  terrain,  friendly  and  enemy  forces  has  great  potential  to 

enable  the  scaling  and  extension  of  DMO  (McVearry  et  ah,  2008). 
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D,  ALTERNATIVE  ARCHITECTURES  FOR  PERSISTENT  VIRTUAL 

ENVIRONMENTS 

There  are  many  research  efforts  directed  towards  developing  alternative 
architectures  for  distributed  virtual  environments.  These  efforts  can  be  generally 
categorized  into  two  groups. 

•  Hybrids  between  client-server  and  peer-to-peer  architectures 

•  Development  of  protocols  specifically  designed  for  distributed  virtual 
environments. 

The  following  are  descriptions  of  representative  examples. 

1.  Brigham  Young  University  (BYU)  Hybrid  Game  Architecture 

The  goal  of  the  hybrid  game  architecture  developed  at  Brigham  Young  University 
(BYU)  is  to  reduce  bandwidth  requirements  for  MMOGs  while  maintaining  central 
control.  Central  control  is  required  in  this  architecture  because  it  is  important  to  control 
the  sequence  of  events  in  the  game,  enforce  the  game  rules,  and  update  the  game  logic 
and  environment.  This  hybrid  architecture  possesses  the  following  conceptual 
architecture. 

•  Only  state-changing  moves  are  processed  by  the  central  game  server 

•  Position  changes  are  distributed  by  clients  on  a  peer-to-peer  level  based 
upon  availability  and  game  world  proximity 

The  guiding  principle  of  the  BYU  system  is  that  only  clients  that  are  proximate  in 
the  game  environment  are  concerned  with  the  most  up-to-date  position  of  one  another. 
Moves  such  as  attempts  to  retrieve  objects  or  take  some  combat  action  are  all  controlled 
by  the  server.  In  summary,  movement  messages  are  distributed  peer-to-peer  to  other 
nodes  that  are  in  the  same  virtual  region.  All  other  messages  are  distributed  client-server 
including  movement  messages  for  nodes  not  in  the  same  virtual  region.  Experimental 
results  show  that  bandwidth  requirements  at  the  server  decrease  as  player  density 
increases  within  a  game  region.  When  player  density  is  low  within  a  game  region,  the 
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central  server  is  overwhelmed  beeause  very  few  movement  messages  are  distributed 
peer-to-peer.  In  other  words,  only  a  small  portion  of  the  workload  is  offloaded  to  the 
elients. 

This  partieular  implementation  of  hybrid  client-server  and  peer-to-peer 
arehiteeture  suggests  that  this  type  of  system  is  suitable  for  DVEs  where  player  density 
within  regions  will  typieally  be  relatively  high  and  is  not  worth  special  implementation 
effort  when  this  is  not  the  case  (Jardine,  2008). 

2,  The  University  of  Pennsylvania  Peer-to-Peer  for  Massively 
Multiplayer  Games 

Efforts  at  the  University  of  Pennsylvania  have  produeed  another  elient-server, 
peer-to-peer  hybrid  arehiteeture  that  uses  a  slightly  different  conceptual  framework. 
Instead  of  having  every  client  send  non-persistent  state  updates  (e.g.,  movement)  to  every 
other  client  that  is  proximate  within  the  game  region,  this  arehiteeture  uses  a  self¬ 
organizing  peer-to-peer  overlay  based  upon  both  the  game  region  proximity  of  entities 
and  the  physieal  network  eonnection  between  the  clients,  whieh  control  them  (Knutsson 
et  ah,  2004). 

The  arehiteeture  automatically  selects  trusted  clients  to  act  essentially  as  servers 
for  complete  game-state  distribution  where  selected  clients  are  proximate  in  the  network 
itself,  not  neeessarily  in  the  game  environment.  This  differs  from  the  system  deseribed 
earlier  in  that  complete  state  is  distributed  albeit  to  a  select  number  of  participants. 
EreePastry,  which  is  an  open-source  version  of  Pastry  (Microsoft,  2009),  a  peer-to-peer 
overlay  and  routing  network  (whose  detailed  description  is  beyond  the  seope  of  this 
thesis)  was  used  to  implement  a  simple  Multi-User  environment. 

Position  updates  are  multicast  every  150  ms  to  replicate  the  behavior  of 
MMORPGs.  Network  simulation  results  with  4000  entities  distributed  aeross  100  to  400 
regions  using  a  single  server  showed  that  message  lateney  is  within  an  acceptable  range 
for  most  MMORPGs.  System  performance  suffered  when  player  density  was  non- 
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uniform  from  region  to  region  (Knutson  et  al.,  2004).  While  this  effort  is  in  its  early 
phases,  it  suggests  an  alternate  type  of  hybrid  architeeture  that  is  worth  considering  for 
developers  of  militarily  useful  DVEs. 

3.  MMOX:  Live  Entity  State  Stream  Protocol 

The  popular  online  virtual  environment  Second  Life  (Linden  Labs)  has  received  a 
great  deal  of  popular  attention.  There  are  over  15  million  registered  accounts  but  little 
authoritative  data  on  actual  ongoing  usage.  Linden  Labs  self-reports  that  on  average 
approximately  38,000  players  are  simultaneously  connected.  Second  Life  is  interesting 
because  it  uses  a  slightly  different  information  exchange  framework  than  typical 
MMOGs.  Analysis  of  Second  Life  network  traffic  shows  that  it  is  atypical  for  MMOGs 
(Kumar  et  ah,  2008).  The  developers  of  Second  Life  do  not  describe  it  as  a  MMOG  and 
in  fact,  it  does  not  conform  to  all  characteristics  such  games  that  are  described  earlier. 

MMOX  is  an  Internet  Engineering  Task  Lorce  (lETL)  standards  development 
project  for  Massively  Multiplayer  Online  applications  of  all  types  (lETL-MMOX,  2009). 
The  MMOX  Live  Entity  State  Stream  (LESS)  Protocol,  which  is  derived  from  the  update 
protocol  used  for  Second  Life,  is  geared  toward  high-rate  virtual  world  object  update. 

The  objective  of  the  MMOX  standard  development  is  to  enable  interoperability  between 
different  DVEs.  It  is  specifically  intended  to  enable  participants  to  move  seamlessly  from 
one  DVE  to  another  with  their  persistent  virtual  persona  or  avatar.  While  the  bandwidth 
requirements  for  LESS  (approximately  100  Kbps)  may  limit  its  utility  for  widely 
distributed  military  specific  DVEs,  this  effort  is  an  important  development  (Rosedale  et 
ah, 2008) 

E.  OTHER  MIDDLEWARE  FOR  PERSISTENT  VIRTUAL 

ENVIRONMENTS 

Quite  apart  from  the  difficulties  of  game  development,  MMOGs  add  the 
challenges  of  distributed-client  connection  handling,  load  balancing,  shared  object 
contention,  and  synchronization  of  data.  Despite  many  project  attempts,  there  has  been  a 
significant  amount  of  attrition  in  this  area,  and  much  past  and  recent  literature  describes 


44 


failed  or  stalled  efforts.  As  a  result  the  emerging  popularity  of  MMOGs  has  led  to  a  lot  of 
interest  in  the  development  of  MMOG-speeifie  middleware  applieations.  In  addition  to 
Projeet  Darkstar  deseribed  earlier,  the  Real-Time  Framework  being  developed  at  the 
University  of  Munster  in  Germany  is  another  sueh  middleware  effort.  There  are  also 
some  efforts  to  apply  the  DoD  HLA  to  the  domain  of  MMOGs.  Projeets  that  are  intended 
to  be  eomplete  MMOG  solutions  sueh  as  Multiverse  are  not  within  the  scope  of  this 
thesis,  though  it  is  recommended  that  readers  compare  this  type  of  software  to  the 
middleware  described  here  (Multiverse,  2009). 

1,  University  of  Munster:  Real-Time  Framework 

The  Real-Time  Framework  (RTF)  project  is  being  developed  at  the  University  of 
Munster  in  Germany  as  part  of  the  edutain@grid  (www.edutaingrid.eu)  project  to  support 
development  of  large  multiplayer  virtual  environments.  Its  goal  is  to  liberate  the 
developer  from  low-level  tasks  such  as  multi-server  communications,  object  migration 
across  servers  and  load  balancing.  The  RTF  provides  a  high-level  communication  and 
computation  middleware  for  single-server  and  multi-server  online  games  (Glinka  et  ah, 
2007).  The  distinction  of  RTF  is  that  is  abstracts  all  of  the  components  of  MMOG 
development  including  the  game  engine,  game  logic,  and  game  state  distribution.  RTF 
supports  three  parallelization  schemes  that  are  common  to  MMOGs;  zoning,  instancing, 
and  replication. 

The  RTF  project  has  not  yet  published  implementation  details  beyond  this  level 
and  it  remains  unclear  whether  or  not  the  implementation  or  source  code  will  be 
publically  available.  Such  availability  is  worth  monitoring  because  it  appears  to  be  well 
supported,  and  if  successful  may  provide  a  useful  platform  upon  which  to  base  scalable 
DVEs  of  various  types. 

2,  Cybernet  OpenSkies  MMOG  Middleware/Runtime  Infrastructure 

While  HLA  is  a  specification,  any  particular  HLA  implementation  is  by  definition 
a  middleware  application  because  it  is  used  to  connect  simulation  applications  to  the 
network/communications  infrastructure.  HLA  is  fairly  complex  yet  most  of  its 
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functionality  is  not  needed  for  game  development.  Cybernet  Systems,  which  has 
developed  complete  HLA  RTIs  and  gateways  for  use  in  federating  military  simulators, 
developed  a  mini-RTI  designed  for  MMOGs  called  OpenSkies  (Figure  9). 


Light  T  raffic-  ■ 
Heavy  T  raffiC' 


Figure  9.  OpenSkies  HLA-based  Architecture  for  MMOGs  (From;  OpenSkies,  2009). 

OpenSkies  is  designed  to  convert  the  peer-to-peer  communications  approach  of 
typical  HLA  implementations  to  an  approach  similar  to  the  client-server  approach 
adopted  by  commercial  games  servers  (Cohen  et  ah,  2004).  The  development  of 
OpenSkies  is  partially  predicated  upon  the  belief  that  the  pure  peer-to-peer  approach  of 
military  distributed  simulation  will  be  discarded  as  joint  military  training  simulations 
grow.  Commercial  game-developer  acceptance  of  OpenSkies  is  not  evident  but  the 
concept  of  alternative  architectures  is  relevant  to  this  thesis. 
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F. 


SH-60B  MISSION  REHEARSAL  TACTICAL  TEAM  TRAINER  (MRT3) 
AS  A  GAME  CLIENT 


ONR  funded  the  development  of  the  SH-60B  Mission  Rehearsal  Tactical  Team 
Trainer  (MRT3)  by  Naval  Air  Warfare  Command  Training  Systems  Division  (NAWC- 
TSD)  to  provide  a  PC-based  training  simulation  for  anti-submarine  helicopter  crews. 

It  was  developed  specifically  with  the  idea  that  PC-based  simulation  might  better 
enable  the  creation  of  communities  of  learners.  It  is  built  primarily  upon  commercial-off- 
the-shelf  (COTS)  hardware  and  software  with  the  addition  of  some  application-specific 
user-interface  hardware. 

MRT3  is  a  collection  of  independent  applications  that  reflect  the  specific  crew 
positions  within  the  SH-60  helicopter  (Figure  10).  In  addition  to  stand-alone  use,  the 
MRT3  has  been  extended  to  participate  in  military  distributed  simulations  using  both  DIS 
and  HLA.  The  SH-60B  MRT3  has  been  incorporated  into  several  distributed 
environments  including  Fleet  Synthetic  Training  (FST)  Exercises  (Gallo  et  ah,  2006). 


Figure  10.  Screen  capture  from  the  Airborne  Tactical  Officer  (ATO)  and  Pilot  Station 
of  the  MRT3  (From;  Gallo  et  ah,  2006). 


Since  a  great  deal  of  time  and  effort  has  already  been  invested  in  developing  the 
MRT3,  and  because  it  uses  standard  specifications  for  connecting  to  military  DVEs,  it 
serves  as  a  good  test  case  for  integrating  existing  simulations  with  alternative  DVE 
architectures. 
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G.  X3D:  OPEN  SOFTWARE  STANDARD  FOR  DEFINING  AND 

COMMUNICATING  REAL-TIME,  INTERACTIVE  3D  CONTENT 

1.  X3D  Conceptual  Framework 

Despite  almost  three  deeades  of  graphics  research,  creating  compelling  distributed 
3D  virtual  spaces  is  usually  problematic  due  to  intense  competition  and  little 
interoperability  between  large  numbers  of  proprietary  commercial  technologies.  The 
Web3D  Consortium  in  a  nonprofit  organization  dedicated  to  the  creation  of  open 
standards,  specifications  and  best  practices  in  order  to  promote  the  evolution  of 
technologies  that  can  help  bring  Web3D  to  the  mainstream. 

Extensible  3D  (X3D)  Graphics  is  the  current  Web3D  standard  for  web-capable 
3D  content.  It  is  an  XML-based  standard  that  is  derived  from  Virtual  Reality  Modeling 
Language  (VRML)  that  was  designed  to  overcome  several  of  the  limitations  of  VRML. 
Since  XML  is  a  standard  for  describing  the  scene  graphs  of  3D  content,  browser  and 
viewer  developers  can  implement  the  actual  rendering  of  these  scenes  in  any  way.  Visual 
simulation  is  one  of  the  target  applications  for  VRML  and  X3D  (Blais  et  ah,  2001; 
Brutzman  &  Daly,  2007). 

2.  Authoring  Tools 

Since  X3D  is  XML  and  text-based,  authoring  can  be  done  in  any  environment. 
Productivity  and  quality  depends  upon  the  availability  of  user-friendly  authoring  tools  of 
which  there  are  several.  A  detailed  survey  is  beyond  the  scope  of  this  thesis,  though 
numerous  tools  and  applications  are  listed  on  the  X3D  Resources  page  (Web3D 
Consortium,  2009).  X3D-Edit  is  presented  as  an  exemplar  because  of  its  support  for  DIS 
ESPDU  functionality.  X3D-Edit  is  the  authoring  tool  developed  and  maintained  at  the 
Naval  Postgraduate  School’s  Modeling  Virtual  Environments  and  Simulation  (MOVES) 
Institute  (Brutzman,  2003).  One  of  its  key  tools  for  DVE  development  is  the  DIS  Player- 
Recorder  (Ligure  1 1),  which  enables  inspection  and  analysis  of  DIS  PDUs  as  they  arrive 
over  the  network. 


48 


Figure  1 1 .  DIS  Player-Recorder  Panel  from  X3D  Edit  3.2 


3,  X3D  and  Distributed  Virtual  Environments 

3D  virtual  scenes  without  interactivity  do  not  meet  the  requirements  of  a  DVE. 
X3D  content  can  be  remotely  manipulated  across  the  network  in  realtime  through  the  use 
of  Script  nodes  (Brutzman  &  Daly,  2007).  More  interestingly,  the  X3D  specification 
includes  the  EspduTransform  node  (McGregor  &  Brutzman,  2008).  This  node  allows 
distributed  control  over  geometry  nodes  via  DIS  entity  state  PDEls.  Visualization  of  the 
state  of  the  entities  in  a  DVE  becomes  relatively  trivial  if  the  remote  applications,  which 
control  them,  produce  ESPDEls,  which  reflect  their  state. 

4,  Scenario  Authoring  and  Visualization  for  Advanced  Graphical 
Environments  (Savage)  X3D  Model  Archive 

The  Scenario  Authoring  and  Visualization  for  Advanced  Graphical  Environments 
(Savage)  archive  is  a  collection  of  composable  X3D  models  designed  to  enable 
straightforward  development  of  web-based  virtual  environments.  The  Savage  model 
collection  is  developed  primarily  by  active-duty  military  students  at  NPS  (Eigure  12). 
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The  Savage  eolleetion  ineludes  Savage  Modeling  and  Analysis  Language  (SMAL) 
tactical  metadata,  which  supports  the  3D  authoring  process  by  embedding  information 
about  a  model  inside  the  model  itself.  The  X3D-Edit  authoring  tool  provides  a  growing 
number  of  support  capabilities  for  building  and  testing  composable-networked  X3D 
models  (Blais  et  al.,  2001).  Additional  models  can  be  added  and  used  from  different 
sources  such  as  the  Army  Model  Exchange  (U.S.  Army  Virtual  Targets  Center,  2009). 


Eigure  12.  Model  of  an  AH-IW  Cobra  developed  by  Maj  C.  B.  Eakey,  USMC 
(https://savage.nps.edu/Savage) 

H,  OPEN-DIS:  OPEN-SOURCE  IMPLEMENTATION  OF  DIS 

Even  more  directly  relevant  to  this  work  is  the  Open-DIS  library.  It  is  a  free,  open 
source  implementation  of  the  DIS  standard  implemented  in  C++,  C#  and  Java.  Open-DIS 
and  SAVAGE  models,  combined  with  an  application  that  provides  game  services  such  as 
Project  Darkstar  together  provide  a  relatively  straightforward  means  to  implement  a 
distributed  virtual  environment  that  may  integrate  DIS  datastreams  from  multiple 
sources. 

Open-DIS  is  a  collection  of  applications  and  libraries  to  support  use  of  DIS 

developed  at  the  MOVES  Institute  at  NPS.  Its  ultimate  goal  is  to  make  available  a  full 

implementation  of  the  DIS  protocol  in  multiple  programming  languages  for  multiple 

platforms.  It  carries  a  Berkeley  Software  Distribution  (BSD)  license,  which  means  that 

applications  developed  with  it  (or  modifications  to  it)  are  not  required  to  be  released 

themselves  as  open-source  (McGregor  &  Brutzman,  2008). 
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In  both  C++  and  Java,  the  protocol  data  unit  (PDU)  is  modeled  as  a  class. 
Complete  program  representation  of  all  DIS  PDUs  is  a  cumbersome  task  requiring  over 
20,000  lines  of  code  (McGregor  &  Brutzman,  2008).  An  extensible  markup  language 
(XML)  like  dialect  is  used  to  structure  the  development  of  PDU  classes.  The  Open-DIS 
PDU  class  objects  are  able  to  marshal  themselves  to  XML  or  serialized  Java  objects.  This 
is  in  addition  to  network  reading  and  writing  of  arrays  of  bytes.  Due  to  this  source-code 
auto-generation  capability  the  Open-DIS  implementation  is  thus  relatively  lightweight. 
Tests  have  shown  that  up  to  1,000  PDUs  per  second  can  be  received  and  processed  on 
commodity  hardware  (McGregor  &  Brutzman,  2008).  Open-DIS  is  also  capable  of 
running  on  mobile  computing  devices  as  well  (Figure  13). 


Elements  of  the  Open-DIS  library  form  an  integral  part  of  several  of  the 
components  developed  in  support  of  this  thesis  work.  The  primary  advantage  derived 
from  using  Open-DIS  is  that  it  was  not  necessary  to  develop  an  application-specific 
protocol.  This  allows  the  thesis  work  to  be  extended  to  other  DIS-capable  simulations 
relatively  easily. 


Pdu: 

EntitylnformationFamilyPdu: 

EntityStatePdu, 

FastEntityStatePdu, 

CollisionElasticPdu, 

CollisionPdu, 

EntityStateUpdatePdu 

WarfareFamilyPdu;  FirePdu, 

DetonationPdu 

LogisticsFamilyPdu: 

ServiceRequestPdu, 

ResupplyOfferPdu, 

ResupplyReceivedPdu, 

ResupplyCancelPdu, 

RepairCompletePdu, 

RepairResponsePdu 

SimulationManagementFamilyPdu: 

StartResumePdu,  StopFreezePdu, 

AcknowledgePdu, 

ActionRequestPdu, 

ActionResponsePdu, 

DataQueryPdu, 

SetDataPdu,  DataPdu, 
EventReportPdu,  CommentPdu, 
CreateEntityPdu,  RemoveEntityPdu 


Figure  13.  (1)  Open-DIS  running  on  an  iPhone®  controlling  icon  position,  (r)  A  partial 

sample  of  the  DIS  PDUs  implemented  to  date  in  Open-DIS. 
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I. 


AUTONOMOUS  UNMANNED  VEHICLE  (AUV)  WORKBENCH 


The  importance  of  structured  data  interchange  for  DVE  interoperability  is 
essential.  Currently  individual  Autonomous  Unmanned  Vehicle  (AUV)  systems  typically 
operate  on  their  own  with  a  proprietary  data  format  for  collected  data  and  no  command 
task  language  for  mission  planning  that  is  sharable  among  robots.  The  problem  of  data 
interchange  between  AUVs  is  directly  analogous  to  the  problem  of  DVEs,  which  use 
heterogeneous  clients  (Weekley  et  ah,  2004).  The  AUV  Workbench 
(https://savage.nps.edu/AuvWorkbench)  was  developed  at  NFS  to  provide  a  tool  for 
mission  planning,  mission  rehearsal  and  mission  playback  (Eigure  14). 


Eigure  14.  Screen  capture  from  AUV  Workbench  2D  view  showing  UAV  mission 
waypoints. 


During  the  execution  of  robot  mission  rehearsal  or  robot  mission  playback,  AUV 
Workbench  is  able  to  generate  ESPDUs  that  reflect  the  state  of  the  entities  whose 
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movement  behavior  it  is  representing.  It  supports  underwater,  maritime  surfaee,  airborne, 
and  ground  vehieles.  Individual  vehicle  state  updates  can  be  reported  via  classical  binary 
DIS  PDUs  or  as  DIS-XML  fragments  embedded  in  Extensible  Messaging  and  Presence 
Protocol  (XMPP)  chat  messages.  Visualizations  are  produced  using  networked  X3D 
models  from  the  Savage  archive  (Weekley  et  ah,  2004).  The  AUV  Workbench  is 
therefore  usable  as  another  source  of  entities,  which  can  populate  a  persistent  DVE 
(Brutzman,  1994). 

J.  SUMMARY 

Examples  of  the  usage  of  large  scale  persistent  DVEs  for  knowledge  solicitation 
and  distributed  training  have  been  provided.  Alternatives  to  pure  client-server 
architectures  are  being  examined  for  MMOGs  in  order  to  reduce  the  tendency  of  the 
server  to  become  a  bottleneck.  Since  an  efficient  mechanism  for  state  distribution  is  an 
essential  core  element  for  both  MMOGs  and  large  militarily  useful  DVEs,  there  are 
ongoing  efforts  to  develop  middleware  that  allows  the  developer  to  concentrate  on  the 
application  instead  of  this  core.  Existing  simulators  and  models  can  be  integrated  into  a 
DVE  that  makes  use  of  emerging  technology.  The  Savage  X3D  model  archive  and  related 
applications  such  as  X3D-Edit  and  AUV  Workbench  provide  a  ready  means  to  develop  a 
DVE  application  using  a  DIS-based  middleware  core. 
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IV.  METHODOLOGY 


A,  INTRODUCTION 

Project  Darkstar  can  be  thought  of  as  an  application  server  for  massive 
multiplayer  online  games.  It  is  not  a  game  implementation  but  a  toolkit  to  handle  the 
many  low-level  tasks  that  are  common  to  all  multiplayer  online  games.  The  development 
of  a  DVE  application  that  uses  the  Project  Darkstar  toolkit  has  been  produced  in  order  to 
examine  some  of  the  questions  raised  in  this  thesis.  A  client  simulator  enables  many 
simultaneous  remote  connections  to  be  made  to  the  server  for  testing  purposes.  Gateways 
are  necessary  to  integrate  simulators  that  do  not  use  common  standards  into  a  common 
environment.  Open  standards  are  used  to  exchange  information  and  visualize  the  state  of 
the  DVE.  Performance  and  behavior  measurement  methodology  is  a  fundamental 
component  of  a  systematic  development  process. 

B,  IMPLEMENTATION  OF  A  MMOG  SERVER  APPLICATION 
1,  Project  Darkstar 

a.  Overview 

Project  Darkstar  (PDS)  is  a  free  and  open-source  MMOG  toolkit  from  Sun 
Microsystems  (Project  Darkstar,  2009).  The  server  code  is  distributed  under  the  GNU 
Public  License  (GPL),  while  the  client  code  is  distributed  under  the  Berkeley  Software 
Distribution  (BSD)  license.  While  both  licenses  are  open  source,  this  licensing  choice 
makes  modifications  to  the  server  side  code  “viral,”  and  required  to  be  themselves  open- 
source,  while  client  side  code  can  remain  unconstrained.  Darkstar  is  still  officially  in  a 
pre-release  form  and  lacks  some  critical  release  features,  such  as  multi-server  scalability. 
Darkstar  is  not  yet  a  complete  server-side  implementation  of  game  technology,  but  is 
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nevertheless  a  highly  functional  toolkit  that  allows  the  server-side  component  of  games  to 
be  constructed  easily.  It  can  be  thought  of  as  an  application  server  for  games  (Figures  15 
and  16). 

Project  Darkstar  provides  a  framework  to  handle  common  tasks  such  as 
low  latency  communications,  thread  management,  handling  contention  between  clients 
for  shared  state,  persistence,  and  (in  future  releases)  scalability.  Darkstar’s  basic  insight  is 
to  treat  objects  on  the  server  side  as  persistent,  and  to  make  state  changes  to  them 
transactional  and  parallelizable.  Darkstar  provides  three  main  services  to  the  implementer 
as  subsystems:  a  data  manager,  a  task  manager,  and  a  channel  manager  (Burns,  2007; 
Project  Darkstar,  2009). 

b.  Darkstar  Data  Manager 

The  data  manager  coordinates  access  to  the  persistent,  server-side  objects. 
The  objects  are  stored  in  a  database  and  brought  into  memory  by  the  data  manager;  as  of 
this  writing  the  Berkeley  database  is  used  to  store  the  data  (Olson  et  ah,  1999).  Clients 
may  acquire  read  or  write  access  to  an  object,  and  if  modified  the  object  can  be  written  to 
the  database  by  the  data  manager  in  a  transaction. 

c.  Darkstar  Task  Manager 

The  task  manager  is  responsible  for  scheduling  and  running  tasks,  which 
are  pieces  of  the  server-side  program  that  can  modify  object  state.  Tasks  are  assumed  to 
be  relatively  short  in  duration;  if  they  run  for  more  than  about  100  ms,  the  task  manager 
will  cancel  the  task,  roll  back  any  changes  the  task  made  to  persistent  objects,  and 
reschedule  it  to  run  later.  The  assumption  at  that  point  is  that  the  task  has  been  delayed  or 
blocked  by  some  (hopefully  temporarily)  unavailable  resource.  This  design  choice  is  a 
product  of  experience  in  game  implementations  and  the  nature  of  the  data  manager. 
Project  Darkstar’s  objective  is  to  run  as  many  tasks  in  parallel  as  possible.  To  do  this, 
Darkstar  must  distribute  the  tasks  across  multiple  cores,  multiple  CPUs,  and  eventually 
multiple  hosts;  if  this  can  be  done  efficiently  (particularly  the  latter)  games  can  be  called 
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to  near  infinite  size.  For  this  to  be  praetieal,  the  tasks  must  be  eompletely  independent  of 
each  other,  with  no  dependencies  or  communications  between  the  tasks  (Project  Darkstar, 
2009). 

The  Darkstar  design  therefore  makes  some  assumptions  and  imposes  some 
requirements  about  what  a  “typical”  task  on  a  server  looks  like.  Tasks  are  assumed  to  be 
of  short  duration,  either  prompted  by  a  client  request  or  by  a  server-side  task.  The  typical 
use  case  is  a  client  either  requesting  a  change  to  the  state  of  an  object  or  else  simply 
requesting  the  state  of  an  object,  both  of  which  can  be  handled  in  code  that  does  not  take 
long  to  run.  Programmers  must  design  and  organize  their  tasks  to  conform  to  these 
assumptions  in  order  to  optimize  overall  performance  (McGregor  et  ah,  2009). 


Figure  15.  Call  of  the  Kings  (Gamalocus  Studios)  is  an  in-development  MMORPG  that 
uses  Project  Darkstar  as  MMOG  middleware  (From;  Gamalocus,  2009). 
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When  a  Darkstar  server  fails,  even  ungraeefully  or  unexpeetedly,  upon 
server  reboot  the  task  manager  retrieves  any  ineomplete  or  outstanding  tasks  from  the 
data  store  for  rescheduling.  Persistent  objects  are  likewise  safe  in  the  database.  Thus,  the 
power  plug  on  a  server  running  Darkstar  can  be  pulled,  and  once  rebooted  the  server 
resumes  and  continues  as  it  was  at  the  time  of  failure,  with  all  object  state  preserved 
(Bums,  2007;  McGregor  et  ah,  2009). 

d.  Darkstar  Channel  Manager 

All  communications  with  the  Darkstar  server  must  be  done  via  channels, 
which  abstract  away  the  housekeeping  requirements  of  network  sockets.  As  with  HLA, 
Darkstar  channels  can  define  their  delivery  mechanism  as  reliable  or  unreliable,  ordered 
or  unordered.  Generally,  the  Transmission  Control  Protocol  (TCP)  is  used  unless  the 
channel  is  set  to  unordered  and  unreliable  in  which  case  User  Datagram  Protocol  (UDP) 
is  used.  Darkstar  does  not  define  or  require  any  particular  protocol  between  the  clients 
and  the  server.  Darkstar  simply  passes  data  to  the  programmer-implemented  protocol  as  a 
byte  buffer  wrapped  in  a  Darkstar  channel.  On  the  client  side,  a  relatively  small  amount 
of  code  is  needed  to  implement  the  communications  framework,  and  this  can  be  done  in 
any  of  several  programming  languages  such  as  Java,  C  or  C++,  or  Python  (Project 
Darkstar,  2009).  This  approach  is  quite  powerful  because  it  allows  the  equivalent  use  of  a 
variety  of  client-server  data  protocols  if  messaging  semantics  are  consistent  throughout. 
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Figure  16.  Project  Darkstar  is  a  Sun  R&D  project  for  the  development  of  an  application 
server  for  MMOGs. 


e.  Technology  Roadmap 

Future  planned  features  for  Darkstar  include  the  ability  to  have  multiple 
hosts  that  are  sharing  the  same  data  managed  together  in  an  efficient  way.  This  approach 
is  expected  to  allow  multiple  hosts  to  handle  client  connections  and,  eventually,  load 
balance  Darkstar  tasks  across  cooperating  servers. 

2,  The  MOVES  Darkstar  Server  Architecture 

As  candidate  middleware  for  MMOGs,  Project  Darkstar  (PDS)  is  designed  to 
support  client-server  architectures.  This  is  the  base  case  for  the  MOVES  test 
implementation.  A  simple  server  is  constructed,  which  allows  a  client  to  establish  a 
connection  and  send  messages  to  the  server.  The  messages  are  byte  buffers  and  DIS 
ESPDUs  are  used  in  this  implementation.  The  server  sends  a  copy  of  the  message  to 
every  interested  client.  In  the  base  case,  there  is  no  area  of  interest  management  (AOIM) 
and  so  all  connected  clients  receive  all  messages  (Eigure  17).  The  improved  functionality 
that  this  server  adds  to  a  typical  multicast  DIS  environment  is  that  much  of  the  network 
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programming  is  greatly  simplified  and  abstracted  away.  DIS  simulation  nodes  connect  to 
the  server  via  a  gateway  that  listens  for  incoming  PDUs  on  the  network.  Native  PDS 
clients  are  able  to  connect  directly  to  the  server. 


Figure  17.  Overview  of  the  architecture  of  the  prototype  DVE  that  includes 

autonomous  DIS  nodes  and  Project  Darkstar  clients.  Gateways  are  used  as 
middleware  to  address  heterogeneity  of  clients. 

Given  this  base  case,  an  additional  protocol  is  added  next.  The  X-Plane 
simulation  node  uses  a  gateway  that  translates  its  proprietary  data  format  into  DIS 
ESPDUs.  This  approach  lets  all  DIS-enable  clients  on  the  EAN  listen  to  X-Plane  DIS 
PDUs.  Euture  work  will  construct  a  direct  gateway  between  the  X-Plane  UDP  and  the 
Darkstar  server. 

3.  Adding  Server-side  Game  Logic 

Server-side  entities  increase  the  richness  of  the  environment  and  suggest  the 

notion  of  an  always-on  and  populated  DVE  (Eigure  18).  This  simple  representation  uses 
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the  Open-DIS  library  and  represents  the  server-side  entity  state  as  an  ESPDU.  Dead 
reekoning  or  path  predietion  is  a  DIS  algorithm  that  is  also  eommonly  implemented  in 
online  games.  Server-side  dead-reekoning  of  both  server-side  entities  and  eonneeted 
client  entities  has  been  successfully  implemented  in  this  work  (Figure  19).  This  opens  up 
the  possibility  of  traffic  management  schemes  at  the  server,  and  reconstruction  of  the 
state  of  other  relevant  clients  at  each  connected  client,  using  information  provided  by  the 
server  in  the  event  of  some  fault  at  the  client  (Yasui  et  al.,  2005). 

PDS  is  an  inherently  event-based  system.  Careful  thought  must  be  given  to  which 
processes  are  to  be  executed  as  PDS  tasks  and  which  are  suitably  implemented  as 
function  calls  responding  to  client  requests.  Yet,  other  tasks  can  sometimes  be  delegated 
to  the  clients  themselves.  Of  the  desirable  characteristics  of  a  DVE  system,  scalability 
and  low-latency  state  update  capabilities  are  particularly  useful  as  has  been  discussed 
(Project  Darkstar,  2009).  Proper  division  and  deployment  of  labor  is  essential  to 
effective  DVE  design. 

For  this  work,  the  base  implementation  of  server-side  logic  revolves  around  the 
scheduling  of  so-called  TickEntity Tasks  for  every  entity  at  some  specific  update  rate. 

This  task  is  not  native  to  PDS  but  is  part  of  the  server  implementation  developed  in 
support  of  this  thesis.  Darkstar  tasks  are  either  periodic  or  scheduled.  Scheduled  tasks  can 
either  be  set  to  execute  immediately  or  after  some  delay.  The  baseline  implementation  in 
this  body  of  work  uses  a  periodic  task  to  emulate  the  behavior  of  a  DIS-based  entity  on 
the  server.  The  periodicity  or  tick  interval  is  analogous  to  DIS  heartbeat  update  interval. 
The  effect  of  various  tick  intervals  on  the  server  are  measured  and  examined  in  Chapter 
V.  As  stated,  the  middleware  abstracts  multi-threading  away  from  the  developer.  This 
thesis  is  specifically  concerned  with  appropriate  choices  of  update  handling  when  dealing 
with  large  numbers  of  dynamic  entities. 
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Initialize  (server, properties) 

Declare  DataManager,  ChannelManager,  TaskManager 

for  zero  to  (number  of  starting  server  entities) 

Declare  an  Entity 

Use  the  DataManager  to  get  a  reference  for  the  Entity 

Add  the  Entity  reference  to  the  ScalableHashMap 

Using  the  TaskManager  to  declare  a  task  handler 
Associate  a  HeartbeatTask  with  the  reference 
Associate  a  TickEntityTask  with  the  reference 

Start  the  ClientListener 


Figure  18.  Pseudocode  of  the  server  initialization  logic.  The  periodicity  of  state  updates 
on  the  server  side  is  set  at  this  point. 

It  is  important  to  emphasize  that  running  many  parallel  independent  simulated 
entities  is  not  the  goal  and  is  not  the  difficulty.  Many  independent  threads  can  be 
executed  when  there  is  no  contention  between  objects.  The  scalability  challenge  derives 
from  the  need  to  have  a  consistent  shared  state  between  all  of  the  entities  that  make  up  of 
the  environment  state.  The  entities  must  be  able  to  share  knowledge  if  necessary.  They 
are  not  independent.  This  is  one  of  the  fundamental  distinguishing  requirements  for  a 
distributed  virtual  environment  (DVE). 

The  base  implementation  for  task  scheduling  includes  a  Heartbeat  Task  and  a 
TickEntityTask.  The  heartbeat  task  has  a  5000  ms  (5  seconds)  maximum  periodicity  in 
order  to  satisfy  the  DIS  standard.  Any  entities  for  which  updates  have  not  been  received 
after  five  heartbeats  are  subsequently  removed  from  the  environment  state  by  a  Reaper 
Task.  Future  implementations  may  seek  to  eliminate  the  Heartbeat  Task  for  clients  that 
connect  directly  to  the  server  since  open  socket  connections  provide  an  alternate 
mechanism  for  confirming  that  an  entity  is  live. 


62 


The  TickEntityTask  is  the  meehanism  by  whieh  dead  reekoning  oecurs  for  all 
entities  that  are  registered  with  the  server.  The  time  interval  between  tieks  is  a  test  point 
but  is  defaulted  to  250  ms  in  this  initial  test.  Dead  reekoning  occurs  as  a  simple  function 
call  to  a  member  method  of  the  Entity  object.  Second-order  acceleration-based  dead 
reckoning  is  implemented  though  the  default  server-side  entities  all  have  acceleration 
equal  to  zero. 

P  =  P(0)  +  U*t  +  ^*  A*(0" 

New  Entity  Location  =  Entity  Location  +  Entity  Linear  Velocity  *  time  Step  + 

2 

Vi  *  Entity  Linear  Acceleration  *  (time  Step) 

Eigure  19.  Vector  and  pseudocode  representation  of  second-order  motion  computations 
used  for  dead  reckoning.  Eull  vector  state  includes  both  position  and 
orientation. 

Task  design  and  implementation  is  important  because  of  the  default  task  timeout 
of  100  ms.  Tasks  that  take  more  than  this  duration  due  to  processing  time  or  contention 
with  shared  objects  are  marked  as  failed  by  the  middleware  and  rescheduled.  Such 
failures  are  costly  and  hopefully  avoided  through  proper  overall  design.  Jobs  that  take 
longer  periods  of  time  to  accomplish  (perhaps  due  to  computational  complexity  or 
network  delays)  may  be  better  accomplished  by  a  series  of  tasks  that  cooperatively  start, 
monitor,  and  complete  a  long-duration  job. 

Simultaneous  task  generation  creates  a  likelihood  of  object  contention  and  the 
degree  to  which  this  might  become  a  problem  is  examined  experimentally.  This 
implementation  uses  a  periodic  task  handler  to  schedule  a  Tick  Entity  Task  for  every 
entity  in  the  global  state  with  an  interval  set  upon  server  initialization.  It  follows  that  the 
number  of  tasks  scheduled  per  second  equals  Number  of  Entities  *  (1000  ms/tick 
interval).  Eor  example,  500  entities  with  an  interval  of  250  ms  will  theoretically  result  in 
the  scheduling  of  2000  Tick  Entity  Tasks  every  second.  This  thumb-rule  does  not 
consider  other  less  frequent  tasks  that  are  used  for  server  management.  Eor  instance. 
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simple  dead  reckoning  used  here  is  completed  in  less  than  0.1  ms  on  a  2.6  GHz  x86 
processor  (M  =  0.048  ms,  SD  =  0.014  ms)  and  so  in  itself  does  not  produce  task  failures 
due  to  timeouts. 

Task  failure  does  occur  when  a  task  is  unable  to  obtain  a  lock  on  the  object  it  is 
associated  with  before  the  timeout  deadline  is  reached.  Such  failures  are  easy  to  detect 
and  reschedule.  However,  it  is  difficult  to  analytically  predict  when  such  faults  occur. 
Study  of  the  behavior  of  the  system  under  different  configurations  provides  some  insight 
into  such  problems  and  also  can  suggest  approaches  for  server-side  state-update  logic 
improvements.  For  example,  allowing  tasks  to  fail  only  a  certain  number  of  times  or 
avoiding  the  scheduling  of  dependent  tasks. 

4.  Mixed-Authority  DIS  Entity  State  Server 

A  mixed-authority  server  is  simply  a  system  in  which  the  server  maintains 
authority  over  the  entities  that  it  simulates  internally  while  it  further  accepts  as 
authoritative,  state  updates  received  from  all  remote  clients.  This  is  exactly  in  accordance 
with  the  requirements  of  the  DIS  standard  (IEEE  1278).  As  an  alternative  example, 
server-side  enforcement  of  consistency  might  be  implemented  by  rejecting  inputs  from 
remote  clients  that  do  not  correspond  to  the  server’s  dead  reckoned  record  of  that  entity’s 
state.  This  approach,  however,  is  a  violation  of  the  DIS  assumptions.  Implementing 
deadreckoning  on  the  server  enables  an  architecture  by  which  a  client-server  relationship 
can  be  established  between  DIS  nodes  and  the  Darkstar  server  while  maintaining  the 
autonomy  of  the  DIS  nodes.  All  connections  are  via  the  server.  The  server  creates  an 
entity  whenever  a  new  client  connects  and  begins  to  dead  reckon  the  movement  of  the 
entity  upon  receipt  of  the  first  ESPDU.  The  client  behaves  as  a  normal  DIS  node  except 
that  it  communicates  solely  with  the  server  (Eigure  20).  DIS  logic  can  be  satisfactorily 
supported  as  long  as  server  capacity  and  performance  are  sufficient. 

The  base  implementation  state  distribution  logic  forwards  every  received  state 
update  to  all  clients.  If  there  are  N  clients  connected,  theoretically  this  will  result  in  N 
updates  per  client  at  the  mean  update  rate.  Although  this  is  a  naiVe  estimate  of  the  traffic 
load  at  the  server  node,  it  demonstrates  that  such  a  fully  interconnected  state-distribution 
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method  theoretically  suffers  from  poor  scalability.  For  purposes  of  comparison,  message 
size  is  considered  to  be  equivalent  to  a  DIS  ESPDU  sent  as  a  User  Datagram  Protocol 
(UDP)  packet  of  172  bytes,  (144  bytes  payload  +  28  bytes  header).  If  there  are  500 
remote  clients,  each  of  which  transmits  one  172-byte  update  per  second,  which  is 
forwarded  by  the  server  to  every  remote  client,  the  average  traffic  load  becomes  a 
daunting  43  Mbytes  per  second. 

The  specific  communication  protocol  is  abstracted  away  from  the  implementation 
developer  by  the  Project  Darkstar  middleware.  Inspection  of  common  examples  shows 
that  typically  the  reliable-delivery  TCP  is  used  for  network  traffic.  When  the  server 
receives  a  message  from  the  client,  it  compares  the  update  to  its  internal  model  of  the 
entity.  It  only  forwards  the  message  if  its  model  does  not  match  the  update  within  some 
tolerance.  This  eliminates  the  need  for  transmitting  expected  incremental  changes.  When 
new  participants  log  in,  the  server  initially  provides  each  new  participant  with  the  state  of 
all  the  connected  entities.  Thus,  all  participants  in  the  DVE  share  a  consistent  world 
state,  even  for  latecomers  whose  presence  was  not  known  beforehand. 
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z'  CLIENT  \ 


Figure  20.  Flowchart  showing  the  state  exchange  between  the  client  and  server  for  the 
mixed-authority  DIS  server.  Consistent  state  is  maintained  by  all  existing 
and  arriving  participants. 


66 


5. 


Darkstar  Client  Simulator 


A  DVE  system  is  best  tested  with  actual  distributed  clients  and  human  users.  For 
the  purpose  of  gaining  insight  into  how  to  implement  the  server  logic  and  testing  the 
general  architecture  of  the  system,  it  is  sufficient  to  use  a  client  simulator  that  generates 
many  simultaneous  connections  to  the  server  and  then  sends  messages  that  are  analogous 
to  what  might  be  seen  from  actual  clients  (Kwok,  2006). 

Such  a  basic  client  test  application  was  developed  as  part  of  this  effort.  Platform 
simulation  and  dead  reckoning  behavior  are  added  to  the  package  by  means  of  classes 
from  the  X-Plane  to  DIS  gateway,  which  is  described  below.  Each  simulated  client  runs 
in  its  own  thread,  which  is  instantiated  and  started  by  the  Client  Simulator  (Figure  21). 
The  simulated  clients  attempt  to  establish  a  connection  with  the  remote  server.  Once  a 
network  connection  is  established  each  enters  a  loop  that  executes  its  behavior. 


Figure  2 1 .  Diagram  showing  the  logic  of  the  client  simulator  used  for  testing  the 
connection-handling  behavior  of  the  server. 
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The  test  behavior  consists  of  three-dimensional  (3D)  movement  at  a  random 
speed  from  0  to  100  meters/second.  Initial  direction  and  location  are  randomly  assigned. 
After  a  fixed  interval,  a  turn  is  executed  by  setting  the  acceleration  to  a  suitable  value  for 
a  fixed  interval.  Time  steps  are  50  ms,  which  is  equivalent  to  20  frames  (and 
corresponding  computational  updates)  per  second. 

The  simulated  clients  also  possess  a  dead-reckoning  model  that  is  compared  to  the 
movement  model  at  every  time  step.  If  the  error  between  the  two  exceeds  a  preset 
tolerance  distance  or  orientation,  an  update  message  is  sent  to  the  server  via  the  client 
channel  and  the  dead  reckoning  model  is  updated  with  new  state  values.  This  mimics  the 
behavior  of  a  remote  DIS  simulation  node.  This  step  is  included  because  the  only  purpose 
of  the  simulator  is  to  generate  characteristic  DIS  traffic. 

6.  Project  Darkstar  DIS  Gateway  Client 

Game  logic  in  the  PDS  original  server  implementation  allows  the  server  to 
differentiate  between  entities  that  represent  client  entities  and  server-side  entities.  Game 
logic  can  then  comply  with  DIS  rules  about  the  authority  of  autonomous  nodes.  For  use 
in  DIS  multicast  environments,  a  DIS-Gateway  client  is  built,  which  listens  for  DIS 
PDUs  and  sends  them  to  the  PDS  Server  application  via  a  connection  channel.  The 
gateway,  therefore,  acts  as  a  point  of  aggregation,  which  can  also  segregate  the  DIS  peer- 
to-peer  architecture  from  the  client-server  architecture  of  PDS.  Such  a  flexible  scheme  is 
quite  useful  for  building  hybrid- architecture  DVEs. 

C.  COMPONENT  GATEWAY  DEVELOPMENT 

To  make  use  of  existing  simulations,  gateways  are  developed  as  proof-of-concept 
of  connecting  heterogeneous  DVEs  using  the  middleware. 

1.  Laminar  Research  X-Plane  ®  to  DIS  Gateway 

X-Plane  is  a  popular  commercial  PC-based  flight  simulator  that  dominates  what 
might  be  considered  the  hard-core  PC  flight  simulator  community.  Its  primary  utility  is 
the  built  in  menus  for  sending  messages  about  the  state  of  the  aircraft  being  represented 
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to  network-connected  third-party  applications.  For  this  reason,  it  often  used  for  other- 
than-entertainment  flight  simulation  purposes  (Laminar  Research,  2009).  A  DIS  gateway 
was  constructed  in  this  project  in  order  to  include  X-Plane  as  a  heterogeneous  client. 

There  are  several  types  of  messages,  which  X-plane  is  capable  of  producing  and 
these  are  generally  described  as  X-Plane  UDPs.  The  X-Plane  UDPs  follow  a  proprietary 
format  that  is  described  in  the  documentation  that  comes  with  the  software  (Laminar 
Research,  2007).  The  “DATA”  UDP  contains  all  of  the  information  needed  to  construct  a 
DIS  ESPDU  except  for  some  details  about  the  entity  type,  which  can  be  obtained  from 
the  “SNAP”  UDP. 

The  gateway  process  consists  of  a  loop,  which  listens  at  the  IP  address  and  port  to 
which  X-plane  is  set  to  transmit.  Upon  receipt  of  an  X-Plane  UDP,  it  is  interpreted  and 
the  relevant  data  is  read  into  an  Open-DIS  ESPDU  object.  Since  X-Plane  is  a  real-time 
platform  level  simulation,  it  is  capable  of  generating  messages  at  very  high  data  rates. 
Second-order  dead  reckoning  with  user-selectable  error  tolerance  occurs  at  the  gateway, 
and  only  updates  which  are  not  within  this  distance  tolerance  (as  compared  to  the  dead¬ 
reckoning  model)  result  in  the  transmission  of  a  DIS  ESPDU  via  the  user-selectable 
output  channel.  Eigure  22  is  a  depiction  of  the  data-handling  logic  of  this  process. 
Second-order  dead  reckoning  is  usually  sufficient  to  reduce  output  packet  rate  to  the 
heartbeat  interval.  Part  of  the  launch  panel  for  this  gateway  application  is  shown  in 
Eigure  23. 

Input  rates  as  high  as  40  X-Plane  UDPs  per  second  are  handled  with  little 
apparent  difficulty.  With  dead  reckoning  implemented  and  the  aircraft  in  X-Plane  in 
straight  and  level  flight,  output  rates  are  comparable  to  typical  platform-level  DIS 
entities. 

Several  opportunities  for  improvement  exist.  There  is  a  free  open-source  X-Plane 
Software  Development  Kit  (SDK),  which  can  allow  implementation  of  a  more 
sophisticated  X-Plane  to  DIS  gateway  (X-Plane  SDK,  2009).  More  importantly,  the  need 
for  gateway  translation  might  be  eliminated  completely  by  implementing  customized  X- 
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Plane  data  channels  directly  on  the  Darkstar  server.  At  that  point,  no  translation  is 
needed  between  X-plane  and  DIS  clients,  since  the  data  channels  map  each  protocol  to 
consistent  server-side  semantics  for  position,  orientation,  velocity,  accelerations,  etc. 


X-Plane  uses  a  proprietary  message  format  for  data: 

Four  byte  message  type  header  +  One  empty  byte  +  (A/  x  (4  byte  index  +  8  x(4  byte  data  element))). 
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Figure  22.  Diagram  of  the  X-Plane  to  DIS  Gateway  Process  Logic. 
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Figure  23.  Prototype  X-Plane  to  DIS  gateway  user-interfaee. 

2.  NAVAIR  SH-60B  MRT3  Client 

The  SH-60B  MRT3  already  has  native  support  for  DIS  built  into  the  pilot  station 
of  the  system.  The  HLA  implementation  is  on  the  Instructor-Operator  Station  and  it 
interfaces  with  the  Common  Distributed  Mission  Training  System  (CDMTS)  to  populate 
the  virtual  environment  of  the  MRT3  with  entities.  The  MRT3  is  actually  a  small  but  self- 
contained  tightly-coupled  distributed  virtual  environment  where  each  of  the  crew  stations 
is  a  node  that  communicates  with  the  other  four  nodes  over  a  LAN. 

A  simple  java  client  was  developed,  which  listens  for  the  ESPDUs  that  the  pilot 
station  multicasts,  and  then  either  forwards  them  to  a  local  multicast  address  or  else 
connects  to  the  PDS  DIS  server  and  sends  the  message  to  the  channel.  This, 
unfortunately,  is  a  one-way  interaction  since  the  MRT3  client  only  sends  DIS  packets  and 
does  not  listen  for  DIS  inputs.  Full  interaction  with  the  current  MRT3  system  requires 
implementing  an  HLA  bridge  to  the  Darkstar  server.  This  was  not  done  but  remains  an 
excellent  opportunity  for  future  research.  Special  care  must  be  taken,  however,  since 
independent  HLA  implementations  are  not  required  to  support  data-exchange 
interoperability  with  other  HLA  implementations. 


71 


D,  VISUALIZATION  SCENE  DEVELOPMENT 


1.  X3D  Scene  Authoring 

In  addition  to  using  existing  simulations  as  clients,  it  is  useful  both  for  purposes 
of  visualization  and  for  demonstrating  the  potential  use  of  X3D  to  provide  interaction  in  a 
DVE  built  upon  a  Project  Darkstar  Server.  One  of  the  considerations  is  that  DIS  uses 
geocentric  Cartesian  coordinates  to  represent  entity  location.  X3D  uses  a  coordinate 
system  oriented  towards  the  presentation  of  3D  scenes  on  a  screen.  This  requires  a 
transformation  at  some  point  in  the  process,  typically  performed  by  the  standardized  X3D 
ESPDU  transform  node.  In  some  cases,  local  coordinate  systems  are  used  by  both  X3D 
scenes  and  DIS-enabled  applications. 

Scene  construction  itself  is  the  construction  of  a  X3D  scene-graph  that  contains 
X3D  Earth  terrain  and  entity  models.  The  models  are  added  as  a  node  within  an 
EspduTransform  within  the  X3D  scene  file.  The  terrain  and  the  models  are  used  directly 
from  the  Savage  model  repository  without  modification. 

2.  Keyhole  Markup  Language  (KML)  Document  Construction 

In  order  to  further  demonstrate  the  utility  of  open  standards,  the  Open-DIS  library 
is  used  with  the  Document  Object  Model  (DOM)  API  to  generate  XML  documents, 
which  represent  the  state  of  entities  in  the  virtual  environment.  DOM  is  a  supported 
component  API  of  the  Java  API  for  XML  Processing  (JAXP).  Keyhole  Markup 
Language  (KML)  is  an  XML  language  focused  on  geographic  visualization,  including 
annotation  of  maps  and  images  (OGC,  2008).  Geographic  visualization  includes  not  only 
the  presentation  of  graphical  data  on  the  globe,  but  also  the  control  of  the  user's 
navigation  in  the  sense  of  where  to  go  and  where  to  look. 

At  some  user-selected  periodicity,  this  application  generates  a  KML  document 
from  the  ESPDUs  of  the  entities  about  which  the  client  is  aware  (Ligures  24,  25  and  26). 
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This  allows  the  observation  of  the  position  of  the  entity  in  an  earth-browser  via  the 
network  link  funetion.  While  this  applieation  is  somewhat  ancillary  to  this  work,  is  has 
significant  implications  for  the  further  visualization  of  large-scale  DVEs. 


Figure  24.  Diagram  showing  data  mapping  from  the  DIS  ESPDU  to  Open  Geospatial 
Consortium  (OGC)  Keyhole  Markup  Eanguage  (KME)  standard. 


<?xml  version="1.0"  encoding="UTF-8"  standalone="no"?> 

<kml  xmlns="http://www.opengis.net/kml/2.2"> 

<Document> 

<Style  id="airplaneIcon"> 

<IconStyle> 

<Icon> 

<href>http://maps. google. com/mapfdes/kml/shapes/airports.png</href> 

</Icon> 

</IconStyle> 

<LineStyle> 

<width>2</width> 

</LmeStyle> 

</Style> 

<Placemark> 

<name>Entity  l_0</name> 

<styleU  rl>#airplaneIcon</  styleU  rl> 

<Point> 

<altitudeMode>absolute</altitudeMode> 

<coordinates>-5.35002412341, 36.1 1826004028321, 813. 855103126727</coordinates> 
</Point> 

</Placemark> 

</Document> 

</kml> 


Figure  25.  Representative  KME  document  generated  by  the  X-Plane  to  DIS  gateway 
developed  in  support  of  this  thesis  work. 
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Figure  26.  Screen  capture  showing  a  KML-based  visualization  in  an  Earth-browser  of 
the  entities  for  which  the  client  has  received  state  information. 

E.  SUMMARY 

Project  Darkstar  is  a  specific  implementation  of  middleware  intended  to  enable 
MMOG/DVE  deployment  without  the  error-prone  complexity  of  developing  the 
communication  and  data-handling  infrastructure.  Some  understanding  of  how  the 
middleware  operates  is  necessary  in  order  to  develop  effective  and  efficient  applications. 
The  need  for  gateways  and  the  importance  of  using  open  standards  for  interoperability 
are  completely  independent  of  the  use  of  any  specific  MMOG/DVE  middleware.  Testing 
methodology  includes  monitoring  the  internal  operation  of  the  system  and  the 
communication  among  the  distributed  components.  A  successful  example 
implementation  is  presented  for  testing  purposes.  Test  results  and  analysis  appear  in  the 
next  chapter. 
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V.  EXPERIMENTAL  RESULTS 


A,  INTRODUCTION 

This  chapter  reports  details  of  task  scheduling,  metries  and  measuring 
sueeess/failure  for  various  numbers  of  server-managed  entities.  This  data  provides 
interesting  results  and  raises  questions  about  the  baseline  server-side  logie  for  updating 
the  state  of  managed  entities.  The  eharaeter  of  the  network  traffie  between  the  server 
applieation  and  remote  clients  is  also  reported  and  explained. 

B,  MEASURING  OVERALL  SYSTEM  PERFORMANCE 

1.  Experimental  Parameters  of  Operation 

Understanding  the  behavior  of  the  server  arehitecture  and  logie  requires 
measurement  under  various  eonfigurations.  The  first  study  examines  the  effeet  of  both 
tiek  interval  and  the  number  of  server  side  entities  on  the  behavior  of  the  system.  The 
seeond  study  examines  system  behavior  under  varying  numbers  of  remotely  eonneeted 
simulated  elients.  The  test  hardware  is  a  Dell  PowerEdge  1750  raek-mounted  server 
(Table  2). 

Table  2.  System  Information  for  the  Server  Used  in  this  Work 


Server 

Dell  PowerEdge  1750  raek-mounted  server 

Proeessor(s) 

Two  Intel  Dual-Core  P4/Xeon  2.4  Ghz  ea 

Memory 

1.0  GB 

Operating  System 

Redhat  Enterprise  Einux  version  4.1.2 

Java  Version 

1.6.012 
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2. 


Server  Profiling  Test 


The  server  profiling  experimental  design  is  (7  x  4)  full  factorial.  Entity  behavior  is 
deterministic  and  no  repetition  is  performed. 

a.  Independent  Variables 

•  Simultaneous  Server-Side  Entities:  0,  100,  250,  500,  1000,  1500,  2000 

•  Tick  Interval:  100,  250,  500,  750  milliseconds 

b.  Dependent  Variables 

•  Tailed  Tick  Tasks 

•  Successful  Tick  Tasks 

•  Entity  with  Tick  Task  Contention 

•  Transaction  Conflict  Timeouts 

3,  Network  Traffic  and  Server  Connection-Handling  Test 

Due  to  the  need  to  run  many  simulated  clients  on  multiple  machines,  all  tests  are 
conducted  within  a  100  Mb/s  Ethernet  EAN  in  order  to  avoid  any  routing  or  bridging 
delays.  Each  client  generates  on  the  order  of  a  hundred  bytes  per  second  of  traffic. 
However,  state  updates  received  from  the  server  will  be  many  times  this  amount,  directly 
proportional  to  the  number  of  connected  clients.  Since  all  of  the  simulated  clients  are 
actually  connecting  to  the  same  server  node  this  approach  generates  significant  traffic  at 
that  node.  If  clients  are  distributed  across  a  WAN,  the  requirement  to  operate  inside  of  a 
EAN  does  not  exist.  As  the  tests  were  conducted  within  a  LAN  during  periods  of  low 
traffic,  the  effect  of  network  latency  is  not  examined.  The  tick  interval  is  held  fixed  at 
250  ms  and  the  number  of  server-side  entities  is  kept  at  zero.  The  number  of  remote 
connections  is  varied  through  the  set;  100,  250,  500,  750. 

The  client  simulator  application  is  run  on  two  separate  machines  connected  to  the 
LAN  by  100  Mbs  Ethernet.  Each  is  able  to  generate  500  connections  without  difficulty. 
The  client  simulator  hardware  platforms  are  as  follows. 
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•  Dual-Core  x86  2.2  GHz  Laptop  with  2  GB  RAM 

•  Single-Core  x86  2.666  GHz  Desktop  3  GB  RAM 

a.  Independent  Variables 

•  Remote  Simulated  Clients:  100,  250,  500,  750 

b.  Dependent  Variables 

•  Network  Traffic  between  Server  and  simulator  nodes 

•  Packets  per  second 

•  Packet  Size 

•  Aggregate  Flow 

•  Failed  Tick  Tasks 

•  Successful  Tick  Tasks 

•  Entity  with  Tick  Task  Contention 

•  Transaction  Conflict  Timeouts 

•  Successful  Handle  Channel  Message  Tasks  (Part  of  middleware  package) 

•  Failed  Handle  Channel  Message  Tasks 

4,  Measurement  Methods 

Data  on  the  server’s  behavior  at  different  test  points  is  collected  using  a  profiling 
plug-in  to  the  Darkstar  server  (Darkstar  Profiler,  2009).  Coarse  analysis  provides  insight 
into  the  suitability  of  this  particular  server  implementation.  A  detailed  examination  of  the 
tasks  that  are  scheduled,  succeed,  and  marked  as  failed  points  to  specific  implementation 
weaknesses.  A  network  protocol  analyzer  is  also  used  in  order  to  characterize  the 
communications  traffic  between  clients  and  the  server. 

a.  Profiling  Data 

A  server  profiler  produced  by  an  independent  effort  is  used  to  generate  a 
record  of  each  test  run.  The  record  includes  the  following. 
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•  Successful  and  Failed  tasks 

•  Timeout  exceptions 

•  Object  contentions 

The  profiler  paekage  is  added  to  the  library  for  the  Darkstar  server 
implementation.  The  file  darkstar-server .properties  file  is  modified  so  that  the  profiler  is 
started  coneurrently  with  the  server  (Darkstar  Profiler,  2009). 

The  server  eonfiguration  is  passed  is  also  passed  by  adding  two  properties 
to  the  darkstar-server.properties  file.  These  are  server  entities  and  server  tick  interval.  An 
experimental  run  eonsists  of  editing  these  properties  using  a  text  editor,  starting  the  server 
remotely  with  a  eall  to  an  ant  target  via  an  SSH  client  (PuTTY  is  used)  (Tatham,  2009; 
Apaehe  Foundation,  2009).  The  server  is  first  run  for  one  minute  to  allow  its  behavior  to 
stabilize  after  initialization.  After  an  additional  two  minutes,  the  server  is  stopped  with  a 
eall  to  a  different  ant  target.  The  server  data  store  is  erased  so  that  it  initializes  with  the 
next  set  of  test  parameters  upon  the  next  start.  If  this  step  is  omitted,  the  server  (which  is 
designed  for  reliable  persistenee)  will  start  with  the  configuration  it  had  when  it  was 
stopped  regardless  of  the  darkstar-server.properties  fide. 

The  measured  profiling  data  is  then  downloaded  to  a  loeal  machine  for 
subsequent  analysis.  The  profiler  produces  bloeks  of  data  that  cover  60  seeonds  of  run 
time.  For  tests  with  large  numbers  of  entities,  eaeh  recorded  data  block  may  be  15-20  MB 
in  size. 


b.  Network  Traffic 

The  Wireshark  network  protocol  analyzer  is  used  to  capture  traffic  at  the 
machines  that  are  running  the  elient  simulators  (Wireshark,  2009).  The  address  of  the 
server  is  known  and  so  traffic  from  other  sourees  is  appropriately  filtered  from  the 
capture  trace.  The  aggregate  of  eaptured  traffic  from  the  two  test  nodes  is  the  total 
communication  between  the  server  and  all  of  the  simulated  clients.  Since  the  test  is  done 
in  a  quiet  network  environment,  paeket  loss  is  minimal.  Background  traffic  at  the 
simulator  nodes  is  typieally  less  than  0.1  Mb/s  during  the  tests. 
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The  server  is  started  in  aeeordanee  with  the  proeedure  deseribed  earlier, 
and  then  allowed  to  run  for  one  minute  so  that  it  is  stable  and  ready  to  aeeept 
eonnections.  The  elient  simulator  waits  one  second  between  instantiation  of  simulated 
clients  so  that  the  server  does  not  receive  all  the  connection  requests  simultaneously, 
which  is  a  reasonable  precaution  since  its  ability  to  handle  this  circumstance  is  not  under 
test.  After  the  client  simulator  indicates  that  all  of  its  simulated  clients  are  connected  to 
the  server,  the  traffic  capture  is  initiated  in  the  network  protocol  analyzer  application. 
Five  minutes  of  network  traffic  is  captured  to  smooth  out  any  effects  of  transient 
behavior.  The  capture  file  is  saved,  and  the  server  and  client  simulator  are  each 
reinitialized  and  restarted  with  configuration  parameters  adjusted  according  to  the  test 
design. 


C.  SERVER  PROFILING  WITH  SERVER-SIDE  ENTITIES  ONLY 

1.  Discussion 

Server  profile  behavior  with  zero  server-side  entities  reveals  189  housekeeping 
tasks  over  the  60  second  period  of  data  collection.  As  expected,  there  are  no  task  failures 
or  contentions  at  zero  entities. 

Curiously,  the  server  had  many  more  failed  tasks  with  only  100  entities  than  it  did 
at  all  other  test  levels.  More  usefully,  at  around  500  entities  on  the  server  side,  the  ratio  of 
successful  tick  tasks  to  failed  tick  tasks  reaches  a  maximum.  Recall  that  the  tick  task 
represents  a  state  update  of  each  entity  for  which  the  server  has  a  record.  The  ability  of 
the  server  to  maintain  a  consistent  state  is  determined  by  the  number  of  successful  tasks. 
The  theoretical  number  of  generated  tasks  for  500  entities  at  an  interval  of  250  ms  over  a 
60  second  period  is  120,000.  In  fact,  a  total  of  approximately  14,000  tick  tasks  were 
generated.  Development  of  a  methodology  to  compare  the  state  of  the  server  side  entities 
after  a  run  to  an  analytically  predicted  state  is  required  to  determine  the  effect  of  this 
disparity  and  is  reserved  for  future  work. 

At  2000  server-side  entities,  an  out-of-memory  error  caused  the  server  to  halt 
execution.  No  data  was  collected  for  2000  server-side  entities  (Figure  27). 
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Tick  interval  had  little  apparent  effect  on  the  profiling  data  (Figure  28).  Possible 
explanations  for  this  measured  result  are  that  some  maximum  capacity  is  reached  even  at 
the  least  demanding  test  point  or  that  this  is  a  characteristic  of  the  middleware  package. 
Further  study  is  required. 


Failed  Tick  Tasks  vs.  Tick  Interval  (ms) 


Figure  28.  Effect  of  the  tick  interval  on  task  success  at  the  server. 
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D. 


SERVER  TESTING  WITH  THE  REMOTE  CLIENT  SIMULATOR 


1.  Discussion 

Profiling  of  the  server  exeeution  is  eondueted  for  purposes  of  eomparison  to  the 
server-side  entity  only  data.  Sinee  the  reeord  of  eaeh  elient  is  maintained  as  a  server  side 
entity,  the  profiling  data  is  similar  to  that  of  the  ease  where  there  are  no  remote  elients 
(Figure  29). 
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Figure  29.  (1)  Server  profiling  data  showing  effeet  of  the  number  of  eonneeted  elients. 

(r)  Network  traffic  analysis  showing  effect  of  the  number  of  connected 
clients  on  the  traffic  at  the  server. 
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Since  all  the  clients  are  actually  resident  at  one  node,  all  traffic  between  elients 
and  the  server  is  between  these  two  physical  nodes.  For  purposes  of  this  test,  this  unusual 
distribution  of  clients  is  not  eonsequential.  Of  interest  is  the  ability  of  the  server 
implementation  to  handle  many  dynamic  simultaneous  eonnections.  In  addition,  some 
idea  of  the  network  traffie  generated  by  the  implementation  is  obtained. 

This  implementation  uses  the  Open-DIS  library  to  marshal  ESPDU  objects  to  a 
byte  buffer  (MeGregor  &  Brutzman,  2008).  This  byte  buffer  is  the  actual  message  sent 
from  the  server  to  the  client  using  the  Darkstar  Channel.  Although  a  DIS  ESPDU  is 
typically  144  bytes  in  length,  TCP  paekets  sent  from  each  client  to  the  server  were 
exactly  54  bytes  long  including  the  protocol  header  of  40  bytes  (Tables  3  and  4).  This 
eonfirms  that  the  message  is  streamed  to  the  server  rather  than  sent  as  one  paeket. 
Investigation  reveals  that  the  Projeet  Darkstar  Server  uses  a  communication  setting  of 
TCP-no  delay.  This  setting  disables  nagling,  whieh  is  a  TCP  feature  that  combines 
several  smaller  paekets  into  one  larger  packet  (Technet,  2009). 


Table  3.  Summary  of  network  traffie  generated  during  the  test. 


Number 

of 

Entities 

Total 

(Packets/Sec) 

Total 

(Kbytes/Sec) 

Clients  to 

Server 

(Kbytes/sec) 

Server  to 

Clients 

(Kbytes/Sec) 

Mean  Client 
to  Server 
Paeket  Size 
(Bytes) 

Mean  Server 
to  Client 
Paeket  Size 
(Bytes) 

100 

681.35 

338.74 

18.89 

319.85 

66.31 

806.84 

250 

3645.63 

2057.11 

96.23 

1960.88 

60.73 

951.36 

500 

6612.35 

3900.50 

178.19 

3722.30 

61.84 

997.67 

508 

6979.09 

4143.78 

186.39 

3957.39 

61.72 

999.55 

Table  4.  Summary  of  per  client  packet  rate  and  byte  rate  for  server  to  client  and 
client  to  server  messages  respeetively. 


Number 

of 

Entities 

Server  to  Client 
(Packets/Second/  Client) 

Server  to  Client 
(Bytes/Second/Client) 

Client  to  Server 
(Packets/Second/Client) 

Client  to  Server 
(Bytes/Second/Client) 

100 

3.96 

3198.55 

2.84 

188.93 

250 

8.24 

7843.55 

6.34 

384.93 

500 

7.46 

7444.62 

5.76 

356.39 

508 

7.79 

7790.15 

5.94 

366.92 
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Despite  multiple  attempts  including  running  client  simulators  on  more  than  one 
hardware  platform  simultaneously,  the  maximum  number  of  connected  clients  never 
exceeds  508  even  though  750  clients  attempted  to  connect.  Approximately  ten  percent  of 
packets  sent  from  the  server  to  the  client  simulator  node  are  lost  when  500  and  508 
simulated  clients  are  connected.  No  packets  sent  from  the  client  simulator  to  the  server 
over  established  channels  are  lost.  Inspection  of  the  profiling  data  does  not  suggest  a 
reason  for  the  server  application’s  inability  to  establish  more  than  this  number  of 
connections.  Inspection  of  plots  of  the  traffic  provides  additional  insight. 

100  Simulated  Clients: 

Every  simulated  client  executes  the  same  behavior  model.  Despite  efforts  to 
stagger  the  initialization  of  the  simulated  clients,  strong  periodicity  is  apparent  in  the 
communication  between  the  clients  and  the  server.  Packet  rate  between  the  clients  and  the 
server  are  proximate  as  is  expected.  Inspection  of  the  throughput  in  Bytes/second  shows 
that  each  client  transmission  causes  a  corresponding  surge  in  traffic  from  the  server 
(Figure  30).  Despite  a  mean  traffic  load  of  338  KB/sec,  the  load  peaks  near  1,000  KB/sec 
because  the  simulated  clients  appear  to  be  synchronized.  This  is  consistent  with  the  server 
state  distribution  logic  described  earlier.  The  relative  size  of  the  client-to-server  packet 
and  the  server-to-client  packet  are  comparable  to  MMOGs  discussed  earlier. 
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Figure  30.  Plot  of  the  network  traffie  between  the  server  and  100  simulated  elients. 

Upper  plot  shows  paekets/seeond  whilethe  lower  plot  shows  bytes/seeond. 


250  Simulated  Clients: 


While  the  plot  of  this  traee  tends  to  aggregate  patterns  that  exist  at  small  time 
resolutions,  it  is  apparent  that  inereasing  the  number  of  clients  has  the  effect  of 
smoothing  out  the  traffic  (Figure  31).  This  is  a  logical  result.  In  this  case,  the  peak  traffic 
between  the  server  and  the  client  is  only  slightly  above  the  mean  of  2,057  KB/sec. 
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Figure  3 1 .  Plot  of  the  network  traffic  between  the  server  and  250  simulated  clients. 

Upper  plot  shows  packets/second  whilethe  lower  plot  shows  bytes/second. 

500  Simulated  Clients: 

The  trace  does  not  show  the  stable  network  traffic  behavior  that  was  previously 
observed  at  250  clients  in  Figure  31.  Ten  percent  of  the  packets  sent  from  the  server  to 
the  client  simulator  node  are  lost  and  the  plot  of  the  traffic  resembles  the  familiar  “saw¬ 
tooth”  shape  associated  with  rate  control  in  TCP  (Figure  32).  There  are  no  client-to- 
server  messages  lost  and  90.88%  of  the  lost  packets  are  over  1280  bytes  in  length.  This 
suggests  that  500  entities  is  a  practical  upper  limit  to  the  number  of  simulated  clients  that 
behave  similarly  to  connect  to  this  particular  PDS  DIS  server  implementation. 
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Figure  32.  Plot  of  the  network  traffic  between  the  server  and  500  simulated  clients. 

Upper  plot  shows  packets/second  whilethe  lower  plot  shows  bytes/second. 

750  Simulated  Clients: 

The  current  system  and  test  architecture  are  unable  to  exceed  approximately  500 
simultaneous  connections.  The  platforms  on  which  the  client  simulators  are  running  are 
able  to  generate  the  required  number  of  simulated  clients  however  the  maximum  number 
of  simultaneous  connections  achieved  is  508.  This  experiment  was  attempted  in  two 
ways,  first  with  750  simulated  clients  on  a  single  machine  and  alternatively  with  375  on 
one  machine  and  375  on  another.  In  both  cases,  the  number  of  simultaneous  connected 
clients  reached  exactly  508.  The  loss  rate  of  packets  sent  from  the  server  to  the  clients  is 
10%.  No  packets  sent  from  the  client  simulator  node  to  the  server  were  lost.  The  plot  of 
the  traffic  is  qualitatively  identical  to  the  previous  case  of  500  clients  and  is  not  presented 
here. 
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E,  SUMMARY 

During  the  test  sequence  the  server  implementation  is  stable  and  only  fails  when 
the  number  of  server-side  entities  exceeds  1,500.  However,  inspection  of  the  profding 
data  shows  that  performance  decreases  markedly  when  the  server  must  maintain  state  for 
more  than  500-600  dynamic  entities  when  object  contention  is  possible. 

The  number  of  tick  tasks  generated  was  an  order  of  magnitude  less  than  expected. 
Not  enough  is  understood  about  the  middleware  behavior  to  determine  if  this  is 
acceptable.  A  diagnostic  algorithm  by  which  the  server’s  model  of  the  state  might  be 
compared  against  an  analytically  predicted  state  is  needed  to  test  the  ability  of  the  server 
implementation  to  maintain  a  consistent  world  state. 

One  particular  feature  of  this  implementation  is  a  “Reaper  Task,”  which  removes 
entities  from  the  server  when  no  state  update  is  received  from  a  client  for  more  than  five 
“Heartbeats.”  During  testing  with  either  100  or  more  server-side  entities  or  100  or  more 
connected  clients,  this  task  failed  100%  of  the  time.  An  alternative  method  to  accomplish 
this  behavior  (or  a  debugged  API  method)  is  still  needed. 

The  small  size  of  client-to-server  packets  compared  to  the  size  of  the  DIS  ESPDU 
that  forms  the  body  for  all  messages  to  the  server  introduces  new  questions  about  how  the 
client  communicates  with  the  server  (Figures  33  and  34).  While  one  of  the  goals  of  the 
Project  Darkstar  middleware  is  to  abstract  network  programming  away  from  the 
developer,  some  knowledge  about  the  communication  behavior  is  required  in  order  to  use 
the  middleware  in  a  heterogeneous  environment. 

Network  traffic  grew  geometrically  with  an  increase  in  the  number  of  connected 
clients  from  100  to  250;  however,  it  grew  linearly  when  connected  clients  went  from  250 
to  500.  Alternative  server-side  logic  is  likely  necessary  to  avoid  geometric  growth  in 
network  traffic  over  certain  ranges  of  connected  clients. 

The  server  application  and  test  methodology  described  here  establish  a  foundation 
for  additional  exploration.  In  addition  to  potential  applications  of  this  middleware,  the 
test  results  themselves  have  generated  questions  that  need  to  be  answered  in  order  to 
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fully  determine  if  this  is  a  feasible  approaeh  to  militarily  useful  distributed  virtual 
environment  state  distribution  and  interoperability.  In  general,  within  identified  limits, 
performanee  is  excellent  and  Darkstar  behaves  as  expected. 
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Figure  33.  Packet  length  distribution  for  packets  sent  from  the  client  to  the  server.  The 
mean  packet  size  was  60  bytes. 
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Figure  34.  Packet  length  distribution  for  packets  sent  from  the  server  to  the  client. 


89 


THIS  PAGE  INTENTIONALLY  LEET  BLANK 


90 


VI.  CONCLUSIONS  AND  RECOMMENDATIONS 


A,  TESTING  CONSIDERATIONS 

Since  DoD  distributed  virtual  environments  must  ineorporate  many  heterogeneous 
simulations,  and  beeause  no  pure  arehiteeture  ean  be  applied  with  equal  effeetiveness 
aeross  all  seales,  most  military-useful  DVEs  expeeted  to  be  hybrids.  There  is  no  one  size 
fits  all  arehiteeture  for  sueh  diverse  requirements.  The  growth  of  eommereial  MMOGs 
and  the  eonsequenees  for  poor  system  performanee  means  that  signifieant  resourees  are 
being  applied  toward  the  development  of  these  teehnologies  and  the  tools  that  enable 
them.  As  a  result,  the  eommereial  gaming  seetor  is  likely  to  pass  the  DoD  as  pioneers  of 
large-seale  DVEs.  One  fast-progressing  area  where  DoD  developers  ean  benefit  is  in 
leveraging  the  middleware  that  is  being  developed  for  eommereial  MMOG  applieations. 

Testing  reveals  oeeasionally  undesirable  behavior  of  the  baseline  applieation  used 
in  this  work  and  suggests  alternative  implementations  that  are  expeeted  to  be  more 
effeetive  and  more  effieient.  Onee  these  eore  implementation  issues  are  addressed  and  re¬ 
implemented  satisfaetorily,  this  work  provides  a  ready  foundation  for  a  wide  variety  of 
important  researeh  direetions.  Any  applieation  requiring  interaetivity  and  shared  state 
distribution  among  large  numbers  of  remote  partieipants  is  a  potential  avenue  for  applied 
researeh  using  these  teehnologies. 

B,  CONCLUSIONS 

1.  Most  Militarily-Useful  DVEs  will  use  Hybrid  Network  Architectures 

The  development  of  large-seale  distributed  virtual  environments  is  a  eomplex 
task.  Sinee  military  operations  inherently  involve  many  heterogeneous  elements,  this  task 
is  made  more  eomplex  for  developers  of  militarily  useful  DVEs.  Sueh  diffieulties  are 
further  eompounded  by  the  basie  differenees  between  simulations  and  C4I  systems. 

While  game  developers  must  be  eoneerned  with  heterogeneity  of  the  hardware  upon 
whieh  their  applieations  will  run,  military  DVEs  must  be  eoneerned  with  the  syntaetie, 
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conceptual  and  semantic  interoperability  of  the  eonnected  simulations.  The  DoD 
pioneered  the  development  and  use  of  distributed  virtual  environments  for  real-time 
platform-level  applieations  with  SIMNET  and  the  Distributed  Interaetive  Simulation 
protocol.  Many  early  networked  eomputer  games  borrow  heavily  from  the  eoneeptual 
arehiteeture  of  DIS.  As  early  development  was  for  this  speeifie  applieation  domain,  it  is 
diffieult  to  extend  to  large-seale  seope  other  than  by  using  platform-level  real-time 
distributed  virtual  environments.  HLA  supports  eomposability  of  simulations  at  any  level 
of  abstraetion,  not  just  at  the  platform  level.  While  the  HLA  standard  defines  the 
requirements  for  HLA  eomplianee,  the  performanee  and  interoperability  of  a  partieular 
system  is  dependent  upon  the  eommereial  RTI  implementation  ehosen.  In  faet,  most 
DVEs  implemented  in  the  Defense  seetor  are  arehiteetural  hybrids  as  well  as  hybrids  of 
HLA  and  DIS.  This  will  be  true  for  the  near  future. 

2,  MMOG  Technology  will  Develop  more  Rapidly  than  Defense-Specific 
DVEs 

Despite  the  differenees  between  networked  games  and  militarily  useful  DVEs, 
they  share  in  eommon  the  primary  elements  of  a  distributed  virtual  environment  (DVE). 
Large-seale  militarily  useful  DVEs  and  MMOGs  have  analogous  design  requirements 
sueh  as  area-of-interest  management  and  traffie  management  (AOIM)  through  position 
predietion.  Sinee  MMOGs  are  a  service  and  not  a  product,  they  must  efficiently  and 
effeetively  meet  these  design  requirements  for  an  indefinite  period.  While  the  user- 
interface  is  a  game,  a  MMOG  system  is  a  meehanism  for  distributing  shared  state.  The 
signifieant  growth  of  this  industry  ereates  eompetitive  pressure  to  improve  the 
teehnology.  It  is  likely  to  develop  mueh  more  rapidly  than  military-speeifie  applieations. 
The  ehallenge  is  to  maintain  interoperability  through  standards  while  taking  advantage  of 
these  rapid  developments. 

Persistenee  of  a  DVE  is  the  ability  to  maintain  the  shared  state  for  an  extended 
period  of  time.  A  persistent  environment  provides  an  “always  on”  spaee  for  distributed 
partieipants  to  interaet.  It  also  allows  the  effeet  of  partieipant  aetions  to  permanently 
affeet  the  state  of  the  DVE,  and  for  the  effeets  of  partieipant  interaetion  to  play  out  over 
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days,  weeks,  or  even  months.  Unfortunately,  a  typieal  military-speeifie  distributed  virtual 
environment  is  not  eharaeterized  as  persistent  in  eomparison  to  commereial  MMOGs.  A 
few  days  are  the  maximum  reported  persistenee  of  defense-sector  DVEs.  Arguably, 
persistence  is  a  key  element  in  the  growth  of  MMOGs. 

3.  Middleware  is  a  Key  Component  of  Distributed  Virtual  Environments 

Large-scale  DVE  development  is  inherently  complex.  Pure  client-server  or  peer- 
to-peer  architectures  cannot  be  applied  uniformly  across  all  ranges  of  DVE  applications 
because  networks  were  not  designed  for  distributed  virtual  environments.  Eurthermore, 
many  stand-alone  systems  that  are  usefully  integrated  into  a  DVE  were  not  designed  to 
operate  with  heterogeneous  systems.  Middleware  is  a  system  placed  between  two 
applications  to  deal  with  the  heterogeneity  of  the  two  applications.  Eurthermore, 
middleware  is  placed  to  manage  the  interaction  between  two  perhaps  dissimilar 
applications.  It  is  difficult  to  avoid  the  need  for  middleware  even  when  strong  standards 
are  available  in  use.  In  fact,  because  many  military  DVEs  are  hybrids,  middleware  is 
absolutely  required. 

Commercial  MMOGs  must  not  only  distribute  state  updates  efficiently  but  they 
must  enforce  relevant  consistency  between  clients.  This  must  be  done  in  a  manner  that  is 
seamless  to  users.  When  the  need  to  maintain  a  shared  persistent  environment  is  added,  as 
well  as  requirements  for  authentication  and  failure  handling,  the  complexity  often 
exceeds  the  capabilities  of  most  developers.  Middleware  that  abstracts  the  details  of  this 
functionality  from  developers  while  providing  the  necessary  services  is  a  key  enabler  of 
the  growth  of  large-scale  persistent  DVEs  and  MMOGs. 

4.  DVE  Systems  are  Complex  and  Must  be  Tested  to  be  Understood 

The  interaction  between  the  system  logic,  the  connected  nodes,  and  the  network  is 
difficult  to  foresee.  While  there  may  be  some  insight  gained  from  analytical  models  of 
the  system,  the  abstractions  and  assumptions  required  to  develop  the  model  may  make  it 
difficult  to  extend  the  results  to  aggregate  distributed  behavior.  The  logic  implemented  in 
this  case  for  state  distribution  resulted  in  system  performance  varying  as  the  number  of 
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entities  ehanged,  but  modifying  the  update  interval  has  no  apparent  effeet  on  the  system. 
This  result  was  unexpeeted.  It  is  important  to  note  that  beeause  Projeet  Darkstar  is  an 
applieation  middleware,  this  behavior  may  be  a  bug  or  limitation  partieular  to  this 
speeifie  server  implementation.  Some  other  undiagnosed  pathology  might  also  be  the 
eause. 

System  testing,  whieh  explores  all  of  the  relevant  parameters,  requires  eareful 
design.  The  experienee  of  this  work  demonstrates  that  manual  testing  is  not  feasible  for 
anything  other  than  low  resolution  over  a  few  design  parameters.  The  test  results  do 
provide  some  insight,  whieh  may  enable  the  development  of  future  implementations. 
Furthermore,  eontinuous  persistent  testing  under  a  wide  range  of  eonditions  is  neeessary 
for  robust  analysis  of  performanee  under  expeeted  and  unexpeeted  operational  load. 

C.  RECOMMENDATIONS  FOR  FUTURE  WORK 

The  use  and  value  of  an  open-souree  MMOG  middleware  to  eonneet 
heterogeneous  simulators  is  elearly  demonstrated  by  this  work.  The  test  results  raise  new 
questions  about  the  performanee  and  behavior  of  this  partieular  state  update  and 
distribution  arehiteeture.  The  software  tools  developed  and  the  lessons  learned  suggest 
several  direetions  for  further  useful  researeh. 

1,  Alternative  Server  Logic  for  State  Distribution 

The  test  results  show  that  simultaneous  seheduling  of  state  update  tasks  for  all  of 
the  entities  on  the  server  results  in  many  failed  tasks.  While  these  initialization-failure 
tasks  are  reseheduled  and  ultimately  eompleted,  the  reeovery  delays  ean  lead  to 
ineonsisteney  between  state  updates.  Additionally,  there  is  no  way  to  prediet  when  a 
speeifie  task  will  be  exeeuted.  The  need  to  distribute  the  proeessing  of  state  updates  is  not 
peeuliar  to  the  Projeet  Darkstar  middleware.  It  is  one  of  the  key  ehallenges  of  developing 
sueh  systems.  A  design  in  whieh  an  update  is  seheduled  for  every  objeet  needs  to  be  re¬ 
examined.  Area  of  interest  management  (AOIM)  and  objeet  behavior-update 
eonsiderations  are  possible  approaehes  to  address  this  ehallenge. 
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The  server  logie  has  a  signifieant  effeet  on  the  network  traffic  produced  by  the 
system.  While  clients  drive  the  behavior  of  the  server,  the  response  is  non-linear.  In  fact, 
there  is  no  need  for  the  server  to  distribute  every  received  state  update  to  all  clients. 
Prediction  schemes  already  applied  to  DVEs  need  to  be  applied  as  consistently  as 
possible  for  each  type  of  network  connection  supported.  The  server  only  needs  to  forward 
updates  that  change  the  shared  environment  state.  Alternatively,  a  pull  versus  push 
architecture  might  better  be  used  for  the  server- to-client  communications. 

2.  Consistency  Comparison  of  System  State  to  Distributed  Client  State 

While  inspection  of  the  profiling  data  showed  that  failed  state  update  tasks  were 
rescheduled  and  ultimately  succeeded,  the  number  of  such  tasks  was  an  order  of 
magnitude  lower  than  expected.  This  raises  the  question  as  to  whether  or  not  object  state 
is  being  updated  as  expected.  Without  comparison  to  some  expected  state,  this  is  difficult 
to  examine. 

The  states  of  the  entities  on  the  server,  which  represent  distributed  clients  need  to 
be  compared  to  an  analytical  model  of  the  simulation  environment  as  well  as  the  actual 
clients  themselves.  Distributed  simulation  state  consistency  characterization  is  a  nascent 
research  area  specifically  with  regard  to  DVEs.  The  ability  to  characterize  distributed 
consistent  state  formally  is  an  important  for  systematic  development  of  useful  operational 
systems. 


3.  Two-Way  Interaction  for  Legacy  Simulation  Clients  via  an  MMOG 
Middleware 

The  steps  necessary  to  connect  simulation  clients  to  an  MMOG  middleware  are 
discussed  earlier.  The  extent  of  implementation  in  this  work  did  not  go  beyond  one  way 
communication  from  the  client  to  the  other  clients  and  the  server,  although  it  permits 
visualization  and  measurement.  This  approach  is  incomplete.  To  achieve  a  true  DVE  such 
communications  must  be  two-way.  In  order  to  implement  this,  the  client-specific 
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communication  protocols  must  be  reconstructed  from  the  standard  DIS  protoeol  used  for 
eommunication  in  this  system.  Any  neeessary  coordinate  transformations  must  be 
ineluded  as  well. 

Sinee  many  legacy  platform  simulations  are  designed  for  updates  at  essentially 
display  frame-rate,  the  low  rate  of  updates  from  this  MMOG-like  system  may  require 
message  spawning  at  some  legaey  elients.  No  assumptions  about  the  ability  of  the  legacy 
system  to  handle  updates  at  a  fraetion  of  the  rate  it  was  designed  for  ought  to  be  made,  so 
interface  adapters  may  be  needed. 

The  legaey  simulation  itself  must  have  the  ability  to  diseover  objeets  from  the 
messages  it  reeeives;  otherwise,  all  sueh  interaeting  objeets  will  need  to  be  manually 
inserted  and  a  eommunieation  ehannel  assigned  to  it.  This  may  be,  in  faet,  desirable  if 
only  eertain  distributed  objeets  are  of  interest.  Onee  again,  the  need  is  elear  to  aehieve 
both  syntactie  and  semantie  interoperability  when  establishing  interoperability  among 
dissimilar  systems. 

4,  X3D  and  JAVA  MMOG  Framework  for  Militarily  Useful  DVE, 

Two  of  the  major  ehallenges  in  broadly  expanding  the  use  of  DVEs  for  military 
applieations  are  the  development  and  distribution  of  elient  user  interfaees  that  support 
network  demands.  Almost  all  DVE  applieations  require  the  installation  of  proprietary 
elient  software  for  partieipation.  Information  teehnology  management  polieies  often 
make  this  lieensing  requirement  untenable  for  government  use.  Eurthermore,  the  models 
developed  for  these  proprietary  elients  are  generally  not  usable  for  other  applications  due 
to  their  closed  nature,  laek  of  doeumented  testing  and  possible  eneumbranee  by  hidden 
patents. 

Sinee  X3D  is  the  Web3D  Consortium  standard  for  web-based  3D  graphies,  tools 
and  models  developed  for  any  applieation  are  readily  transported  to  any  other  applieation. 
The  only  requirement  to  interaet  with  a  seene  is  a  suitable  X3D  browser.  Several  dozen 
X3D  browser  implementations  and  importers  are  eurrently  available  ineluding  open- 
souree  versions  and  eommereially  supported  software  paekages.  Eaeh  of  these 
approaehes  is  compatible  with  X3D. 
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X3D  animation  can  be  controlled  by  embedded  networked  seripts  and  also 
directly  by  DIS  entity  state  PDUs.  While  the  traffie  at  the  network  server  node  may  be 
high  when  there  are  many  simultaneous  partieipants,  the  traffie  at  distributed  clients  ean 
be  relatively  low.  Three  Kilobits  per  second  (Kbps)  is  typical  for  such  systems  and 
eomparable  to  the  client  node  traffic  for  MMORPGs.  This  enables  distributed  simulation 
across  even  very  low-bandwidth  networks  provided  the  elient  nodes  are  widely 
distributed. 


5.  Consistency  Enforcement  for  Distributed  Simulation 

Commercial  distributed  game  service  providers  often  use  a  client-server 
connection  model  in  order  to  control  the  quality  of  the  game-play  experience.  This  is 
important  because  users  will  not  maintain  their  paid  subscriptions  if  the  experience  is 
poor.  For  military  applications  of  DVEs,  credible  interactions  are  necessary  for  effective 
participant  immersion.  Armored  vehicles  that  move  as  fast  as  aircraft  or  ignore  the  effects 
of  weapons  fire  (for  example)  reduce  the  face  validity  of  the  system.  For  commercial 
games,  various  forms  of  cheating  are  one  of  the  greatest  disrupters  of  game-play 
experience.  It  is  assumed  that  any  client  may  be  dishonest.  The  opposite  assumption 
holds  for  developers  of  military-application  DVEs.  Even  when  all  clients  are  honest, 
differences  between  (or  weaknesses  within)  specific  simulations  may  have  the  same 
effect  as  cheating,  without  the  malicious  intent. 

Consistency  enforcement  is  the  active  assurance  of  a  shared  state  between  the 
participants  of  a  DVE.  Although  already  difficult  to  implement  in  a  game,  the 
heterogeneous  nature  of  military  DVEs  may  greatly  increase  this  difficulty  because 
knowledge  about  the  internal  logic  and  external  behavior  of  the  client  is  not  always 
known.  Game  developers  control  all  nodes;  developers  of  heterogeneous  DVEs  do  not. 

A  basic  form  of  consistency  enforcement  includes  control  of  interactions  but  not 
the  movement  of  the  client  entity.  This  is  analogous  to  hybrid  DVEs,  which  distribute 
position  updates  in  a  peer-to-peer  fashion  and  interactions  between  entities  in  a  client- 
server  fashion.  The  idea  is  that  the  movement  only  changes  the  shared  environment  state, 
for  that  entity  and  for  others,  which  can  see  that  entity. 
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a.  Army  Research  Laboratory  (ARL)  DIS  Lethality  Server 

An  example  of  eonsisteney  enforeement  in  an  otherwise-autonomous  node 
DVE  is  the  DIS  Lethality  Communieation  Server  developed  by  the  Army  Researeh 
Laboratory  (ARL).  In  this  approach,  a  single  vulnerability/lethality  server  provides 
standard  DIS  damage  states  to  entities.  The  system  is  designed  to  centralize  the 
maintenance  of  computing  damage  state  stables;  thus,  simplifying  the  configuration  of 
DIS  exercises.  One  key  goal  is  to  eliminate  variability  in  lethality  results.  Including  such 
a  server  as  a  federate  in  an  HLA  federation  is  a  logical  consequence  (Sauerborn,  1999). 
There  is  no  conceptual  difference  between  that  server  and  the  system  developed  for  this 
work.  Both  are  means  of  distributing  a  shared  consistent  state. 

b,  NA  VAIR  UA  V/UCA  V Distributed  Simulation  Infrastructure 

Naval  Air  Systems  Command  (NAVAIR)  has  developed  an  unmanned 
aerial  vehicle  (UAV)/unmanned  combat  aerial  vehicle  (UCAV)  distributed  simulation 
infrastructure.  Its  objective  is  to  explore  how  UAVs  and  UCAVs  can  function  together  in 
the  future.  The  element  of  the  infrastructure  that  is  relevant  to  this  work  is  the  use  of  a 
weapons-effect  server  in  a  DIS-based  DVL  (Twesme,  2003). 

A  potential  application  of  MMOG  technology  for  simulated  UAV 
operations  is  as  a  persistent  data-store.  Real-world  objects  or  environments  of  interest 
under  observation  by  UAVs  might  be  updated  over  days,  months,  and  years  in  a 
persistent  virtual  environment. 

6.  Migrating  MOVES  MMOG  Server  to  NFS  Hamming  High 

Performance  Computing  Cluster 

The  current  implementation  runs  on  a  single  server  within  the  LAN  and  500 
simultaneous  connections  and  up  to  1500  dynamic  server-side  entities  were 
demonstrated.  Alternative  state  distribution  logic  may  enable  higher  numbers  but  they 
will  still  be  on  the  order  of  1000  clients  per  server.  The  Naval  Postgraduate  School 
recently  installed  the  hamming  system  for  high-performance  cluster  computing.  The 


hamming  system  is  a  Sun  Microsystems  6048  blade  system  consisting  of  144  blades  that 
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together  eontain  a  total  of  1152  cores.  The  system  has  1 12  terabytes  of  disk  space  (Naval 
Postgraduate  School,  2009).  This  system  provides  an  opportunity  for  the  development  of 
an  NPS  or  DoD  wide  persistent  virtual  environment.  Ten  percent  of  this  capability  could 
theoretically  support  on  the  order  of  10,000  simultaneous  participants.  Recall  that  the 
user  interface  is  defined  at  the  client  level.  Such  participants  are  not  necessarily 
individuals  using  a  computer  to  control  an  avatar  in  a  3D  scene  but  could  be  any 
distributed  device  or  individual,  which  has  a  need  to  share  a  common  state. 

7.  Construction  of  a  Bridge  between  an  HLA  Federation  and  a  MMOG 
Server 

One  of  the  strengths  of  HLA  is  its  conceptual  support  for  simulation 
composability.  More  importantly,  HLA  is  the  declared  standard  of  distributed  simulation 
implementation  in  the  DoD.  A  militarily  useful  DVE  must  be  capable  of  participating  in 
HLA  federations.  The  feasibility  of  composing  a  MMOG  middleware  application  into  an 
HLA  federation  is  an  important  research  direction.  The  OpenSkies  middleware  discussed 
earlier  is  an  effort  in  this  area.  The  open-source  Portico  implementation  of  an  RTI  is  also 
worth  exploring  (Portico,  2009). 

8,  MMOG  Application  as  a  Track  Data  Conversion  and  Distribution 
Hub  for  Command  and  Control 

An  MMOG  is  essentially  a  service  for  distributing  and  maintaining  shared  state 
information  of  a  virtual  environment.  This  is  directly  analogous  to  command  and  control 
systems,  which  distribute  and  maintain  a  virtual  copy  of  the  real-world  state.  The 
consistency  of  that  state  can  never  be  better  or  timelier  than  the  quality  of  the  real-world 
information,  which  feeds  it. 

A  logical  application  of  the  state-distribution  mechanism  of  a  MMOG  middleware 
is  to  provide  a  means  for  distributing  track  data  for  command  and  control.  The  word 
“game”  in  MMOG  may  suggest  that  this  is  a  ridiculous  notion.  Recall,  however,  that 
underneath  the  game  implementation,  large  modern  MMOGs  are  among  the  highest- 
performing  distributed  systems  with  several  having  achieved  over  50,000  distinct 
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simultaneous  participants  (IGDA,  2004).  While  the  consequences  for  system  failure 
cannot  compare  to  the  reliability  requirements  of  military  systems,  they  are  not 
inconsequential.  In  any  event,  the  technical  challenges  are  similar. 

A  basic  demonstration  of  the  utility  of  a  track  distribution  hub  includes  the 
capability  to  convert  data  from  one  format  to  another.  This  sees  the  system  as  a  form  of 
middleware  in  addition  to  being  a  state  distribution  mechanism  providing  DIS  mappings 
to  and  from  other  C4I  languages  is  a  good  start. 

Further  work  can  also  connect  hand-held  devices  such  as  GPS-capable  cell 
phones.  As  with  most  scalable  technologies,  the  most  important  success  metric  is  the 
ease  of  integration  rather  than  the  sophistication  of  the  implementation. 

9.  Establish  Persistent  Darkstar  MMOG  Virtual  Battlespace 

Since  commercial  online  games  now  have  lifetimes  measured  in  years,  without 
interruption,  a  similar  capability  can  be  achieved  for  military  DVEs.  This  effort  can 
address  the  significant  limitations  inherent  in  typically  transient,  ephemeral  military 
simulations.  Ongoing  measurement,  testing,  expansion  and  improvement  will  lead  to 
further  capabilities  and  lessons  learned.  Establishing  public  and  controlled-access  DVEs 
for  simulations  based  on  existing  model  archives  such  as  Savage  and  SavageDefense 
using  X3D  Earth  as  a  backdrop  is  a  relevant  goal. 

10,  Integrating  Effective  Shared  Physics  Governing  Sensor  Interactions 

Entities  in  military  DVEs  not  only  need  to  have  physically  based  motion 
governing  their  travel  through  virtual  space,  but  also  need  high-resolution  physics  for 
sensor  applications.  This  is  essentially  true  for  naval  scenarios  where  the  determination 
of  sensor  propagation  may  be  computationally  expensive  and  require  durations  of  many 
seconds  to  complete.  Additional  work  is  needed  to  accomplish  consistent  coherent 
shared  physics  for  phenomenology  other  than  motion  in  DVEs. 
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APPENDIX.  SOURCE  CODE 


Source  code  developed  in  support  of  this  thesis  is  available  at  URL:  http://open- 
dis.sourceforge.net/Open-DIS.html 

Specific  excerpts  of  source  code  are  presented  below. 


public  void  initialize(Properties  props) 

{ 

logger.log(Level.INFO,  "Started  Persistent  Virtual  World"); 

//  Manages  persistent  objects 

DataManager  dataManager  =  AppContext.getDataManagerQ; 

//  Manages  communications  channels 

ChannelManager  channelManager  =  AppContext.getChannelManager(); 

//  Periodic  tasks  to  run 

TaskManager  taskManager  =  AppContext.getTaskManager(); 

try 

{ 

//  Retrieve  an  existing,  known  entity  from  the  data  manager. 

//initializeO  is  called  only  the  very  first  time 

//  a  datastore  is  created,  when  the  datastore  will  always  be  empty. 

Entity  anEntity  =  (Entity)dataManager.getBinding("ENTITY(0,0,l)"); 
logger.log(Level.INFO,  "Found  existing  entity  object"); 

} 

catch(NameNotBoundException  nnbe) 

{ 

//  We  have  a  brand  new  object  store  that  is  completely  empty.  Add  some  objects  and  tasks  to  it. 
logger. log(Level. INFO,  "Creating  initial  server-side  entities  and  tasks"); 

//  The  hash  map  contains  a  key  of  a  string  in  the  format  of 

//  ENTITY(x,y,z),  using  the  entitylD  of  the  DIS  entity, 

entitiesMapRef  =  dataManager.createReference(new  ScalableHashMapO); 

ScalableHashMap  entitiesMap  =  entitiesMapRef  get(); 

//  Save  the  hash  map  of  entities  in  the  system  under  a  named  string 
dataManager.setBindingC'ENTITIES,”  entitiesMap); 

//Get  the  server  side  entity  count  from  the  properties  file 
try{ 

serverSideEntityCount  =  Integer.parseInt(props.getProperty("server.entities")); 

}  catch  (NumberFormatException  nfe){ 

System.out.println("Unable  to  Parse  server.entities  value  using  default  of  "  +  serverSideEntityCount); 
}//end  catch 


Source  code  excerpt  showing  server  initialization  code  of  the  Darkstar  Server 
imnlementation  used  for  this  work. 
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for(int  idx  =  0;  idx  <  serverSideEntityCount;  idx++) 

{ 

Entity  anEntity  =  new  EntityQ; 
anEntity.lastPduReceived.getEntityID().setSite(0); 
anEntity. las  tPduReceived.getEntityID().setApplication(0); 
anEntity. lastPduReceived.getEntityID().setEntity(idx); 

anEntity  .currentState.getEntityID().setSite(0); 
anEntity  .currentState.getEntityID().setApplication(0); 
anEntity  .currentState.getEntityID().setEntity(idx); 

//Set  a  random  initial  velocity  and  position  for  each  entity  with  max  in  any  dimension 
anEntity.  setEntityRandomVelocityO; 
anEntity.  setRandomEntityStartLocation(); 

anEntity  .controlLocation  =  Entity  .ControlLocation. SERVER; 

ManagedReference<Entity>  anEntity  Ref  =  dataManager.createReference(anEntity); 

//  Add  other  information,  such  as  entity  type.  We  really  should  be  reading 
//  info  from  a  config  file. 

//  Put  the  entity  into  a  list  of  entities.  Should  this  be  a  reference  being  added? 

entitiesMap.put(anEntity.toString(),  anEntity); 

//  Instruct  the  scheduler  to  run  a  heartbeat  task  for  this  entity,  which 
//  will  send  out  an  ESPDU  at  a  given  interval  on  the  DIS  channel. 

//  We  set  the  task  to  contain  a  handle  to  the  task 

//  itself  If  the  underlying  entity  object  is  deleted— perhaps  because  it  hasn't 

//  been  heard  from,  or  some  other  reason— the  task  will  cancel  itself  when  it  discovers 

//  the  object  missing  in  the  task's  run()  method. 

HeartbeatTask  heartbeatTask  =  new  HeartbeatTask(anEntityRef); 

PeriodicTaskHandle  heartbeatHandle  =  taskManager.schedulePeriodicTask(heartbeatTask,  10000, 
HEARTBEATFREQUENCY); 

heartbeatTask.  setTaskHandle(heartbeatHandle); 

//  Tick  frequency.  Mostly  a  test  of  object  contention.  The  tick  method  in  an 
//  entity  gets  called  every  DEFAULT_T1CK_1NTERVAL  ms. 

//  A  task-per-entity  is  a  somewhat  suspect  choice  as  you  go  to  more  and 
//  more  entities.  It  might  be  better  to  have  one  tick  task  that  simply 
//  cycles  through  the  list  of  all  entities.  However,  calling  tick() 

//  on  all  the  entities  may  take  a  while,  longer  than  the  task  scheduler 
//  is  willing  to  give.  So  it's  a  tradeoff. 

//Get  the  tick  Interval  from  the  properties 
tty{ 

ticklnterval  =  Long.parseLong(props.getProperty("  server,  ticklnterval")); 

}  catch  (NumberFormatException  nfe){ 

System.out.println("Unable  to  Parse  server.ticklnterval  value  using  default  of  "  +  ticklnterval); 

}//end  catch 

TickEntityTask  tickEntityTask  =  new  TickEntityTask(anEntityRef); 

PeriodicTaskHandle  tickTaskHandle  =  taskManager.schedulePeriodicTask(tickEntityTask,  10000,  ticklnterval); 


Code  excerpt  showing  task  scheduling  in  the  Darkstar  server  implementation. 
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switch(dataType) 

{ 

case  FRAME  RATE: 
dataReader.skipBytes(32); 
break; 

case  ANGULARVELOCITY: 

VectorSFloat  angVel  =  deadReckoningParameter.getEntityAngularVelocity(); 
ang  Vel.  setX(dataReader.  readF  loat()) ; 
angVel.setY(dataReader.readFloat()); 
ang  Vel.  setZ(dataReader.  readF  loat()) ; 

//Set  the  angular  velocity  dead  reckoning  parameter 
deadReckoningParameter.setEntityAngularVelocity(angVel); 

dataReader.skipBytes(20);  //  12  bytes  read,  32  bytes  in  group,  20  left  to  read 
break; 

case  ATTITUDE: 

Orientation  att  =  espdu.getEntityOrientation(); 

att.setTheta((float)Math.toRadians(dataReader.readFloat())); 

att.setPhi((float)Math.toRadians(dataReader.readFloat())); 

att.setPsi((float)Math.toRadians(dataReader.readFloat())); 

dataReader.skipBytes(20);  //  12  bytes  read,  20  bytes  left 

//Actually  set  the  espdu  to  the  decoded  values 
espdu.setEntityOrientation(att); 

break; 

case  LOCATION: 

Vector3Double  loc  =  espdu.getEntityLocation(); 

loc.setY((double)  dataReader.readFloat()); 

loc.setX((double)  dataReader.readFloat()); 

loc .  setZ(FEET_T 0_METERS  *  (double)  dataReader.  readF  loat()) ; 

loc .  convertLati  tudeLongitude  AltitudeT  oD  is() ; 

if  (DEBUG)  System.out.println("x="  +  loc.getX()  +  +  loc.getY()); 

dataReader.skipBytes(20);  //  12  bytes  read,  20  bytes  left 

espdu. setEntityLocation(loc); 
break; 

case  VELOCITY: 

Vector3Float  linearVel  =  espdu.getEntityLinearVelocity(); 
dataReader.  skipBytes(  12); 

linearVel.  setY(dataReader.readFloat()); 
linearVel.  setZ(dataReader.readFloat()); 
linearVel.  setX(dataReader.readFloat()); 

dataReader.skipBytes(8);  //  12  +  12  =  24  bytes  read,  8  left  out  of  32 

espdu.  setEntityLinearVelocity  (linearVel); 
break; 


X-Plane  to  DIS  Gateway:  data  packet  parsing  code  excerpt. 
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